20,741
edits
m (→Motivation) |
|||
Line 35: | Line 35: | ||
=== Scraping === | === Scraping === | ||
{{Note|This will download roughly 4b of data in ~17k files, for each AIRAC cycle!}} | |||
Alternatively, use a media pipeline <ref>http://sergeis.com/web-scraping/downloading-files-scrapy-mediapipeline/</ref> | Alternatively, use a media pipeline <ref>http://sergeis.com/web-scraping/downloading-files-scrapy-mediapipeline/</ref> | ||
<syntaxhighlight lang="python"> | <syntaxhighlight lang="python"> |