Scrapy

Scrapy is a Web Scraping library used by python developers to build scalable web crawlers. It is a complete web crawling framework that handles all the functionalities that make building web crawlers difficult such as proxy middleware, querying requests among many others.