开发技术 - 爬虫基础知识点

为什么选择 Scrapy:

• 优秀的文档 [11]
• 活跃的社区 [12]
• 入门难度低 [13]
• 可扩展性强 [14]
• 并发能力强

100 Hours

框架组件部件和工作流程 [14]

如何实现一个爬虫

• 创建项目 carnie

  $ scrapy startproject carnie
  $ scrapy genspider testspider
  

• 添加爬虫代码 carnie/spiders/testspider.py

  from scrapy import Request, Spider
  
  class TestSpider(Spider):
      name = "testspider"
      start_urls = ["http://www.baidu.com"]
  
      def parse(self, response):
          print response.body
  

• 运行爬虫:

  $ scrapy crawl testspider
  

如何使用代理

yield Request(url, meta={"http": "http://proxyhost:proxyport/"})

可扩展组件：

• Downloader middleware
• Spider middleware
• Item Pipeline
• Extension

如何定时运行：

• 使用 crontab
• 使用 Scrapyd
• 使用 Scrapy Cloud

扩展组件参考代码

• Scrapy 源代码
• Scrapy 官方插件仓库

相关项目

• Scrapyely - A library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely constructs a parser for all similar pages.
• Portia - A tool that allows you to visually scrape websites without any programming knowledges required. With Portia you can annotate a web page to identify the data you wish to extract, and Portia will understand based on these annotations how to scrape data from similar pages.
• Splash - A Javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python using Twisted and QT.
• Scrapyrt - A HTTP server which provides API for scheduling Scrapy spiders and making requests with spiders.
• Crawlera - It allows you to crawl quickly and reliably, managing thousands of proxies internally, so you don't have to.