1. 安装scrapy
代码语言：javascript复制pip install scrapy2. 第一个爬虫代码myspider.py
代码语言：javascript复制import scrapy
class BlogSpider(scrapy.Spider):
name = 'blogspider'
start_urls = ['https://blog.scrapinghub.com']
def parse(self, response):
for title in response.css('h2.entry-title'):
yield {'title': title.css('a ::text').extract_first()}
for next_page in response.css('div.prev-post > a'):
yield response.follow(next_page, self.parse)3. 运行
代码语言：javascript复制scrapy runspider myspider.py -o result.json4. 运行结果
代码语言：javascript复制017-08-06 17:44:56 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
2017-08-06 17:44:56 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2017-08-06 17:44:56 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
...
2017-08-06 17:44:56 [scrapy.core.engine] INFO: Spider opened
2017-08-06 17:44:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-08-06 17:44:56 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-08-06 17:45:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com> (referer: None)
2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>
{'title': u'Scraping the Steam Game Store with Scrapy'}
2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>
{'title': u'Do Androids Dream of Electric Sheep?'}
...
2017-08-06 17:45:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com/page/11/>
{'title': u'Hello, world'}
2017-08-06 17:45:12 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-06 17:45:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2933,
'downloader/request_count': 11,
'downloader/request_method_count/GET': 11,
'downloader/response_bytes': 123915,
'downloader/response_count': 11,
'downloader/response_status_count/200': 11,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 8, 6, 9, 45, 12, 168728),
'item_scraped_count': 105,
'log_count/DEBUG': 117,
'log_count/INFO': 7,
'memusage/max': 43909120,
'memusage/startup': 43909120,
'request_depth_max': 10,
'response_received_count': 11,
'scheduler/dequeued': 11,
'scheduler/dequeued/memory': 11,
'scheduler/enqueued': 11,
'scheduler/enqueued/memory': 11,
'start_time': datetime.datetime(2017, 8, 6, 9, 44, 56, 752503)}
2017-08-06 17:45:12 [scrapy.core.engine] INFO: Spider closed (finished)生成结果的文件result.json
代码语言：javascript复制[
{"title": "Scraping the Steam Game Store with Scrapy"},
{"title": "Do Androids Dream of Electric Sheep?"},
{"title": "Deploy your Scrapy Spiders from GitHub"},
{"title": "Looking Back at 2016"},
{"title": "How to Increase Sales with Online Reputation Management"},
{"title": "How to Build your own Price Monitoring Tool"},
...
{"title": "Spoofing your Scrapy bot IP using tsocks"},
{"title": "Hello, world"}
]解析运行过程
当你运行下面命令时，Scrapy框架会启动爬虫引擎，根据myspider.py中的逻辑进行抓取网页，然后把结果存到result.json中。
代码语言：javascript复制scrapy runspider myspider.py -o result.json第一步：爬虫先请求start_urls中定义到URLs。本例中，只有一个URL。请求该URL返回内容，如下所示。
代码语言：javascript复制...
<h2 class="entry-title">
<a href="https://blog.scrapinghub.com/2017/07/07/scraping-the-steam-game-store-with-scrapy/" rel="bookmark">Scraping the Steam Game Store with Scrapy</a>
</h2>
...
<h2 class="entry-title">
<a href="https://blog.scrapinghub.com/2016/10/06/interview-how-up-hail-uses-scrapy-to-increase-transparency/" rel="bookmark">Interview: How Up Hail uses Scrapy to Increase Transparency</a>
</h2>
...
<div class="col-md-6 prev-post">
<a href="https://blog.scrapinghub.com/page/2/">OLDER POST<i class="fa fa-angle-double-right"></i></a>
</div>
...第二步：调用callback方法parse，解析返回到内容。在回调函数parse中，爬虫循环使用CSS选择器 h2.entry-title，找出网页中的<h2 class="entry-title">元素，并这些元素中的text找出来，生成Python的dict格式数据。
当前页中所有的h2.entry-title元素找完以后，爬虫会执行下面的代码
代码语言：javascript复制for next_page in response.css('div.prev-post > a'):
yield response.follow(next_page, self.parse)它会找div.prev-post元素，然后再次递归执行parse方法。从网页上可以看到，div.prev-post元素是翻页链接，所以，爬虫会不断翻页，知道抓取所有的网页。
第三步： 生成的字典数据存到result.json文件中
scrapy的任务是异步执行的，也就是说，它不用等一个请求返回以后才发送另一个请求，而是可以同时进行的。这可以加快运行速度。我们也可以对Scrapy进行设置，比如每一个请求延迟一段时间，等等。
