Cobweb
на сайте с 15 декабря 2022, 14:04
web crawler that can use resque to cluster crawls to quickly crawl extremely large sites.
Web crawler with very flexible crawling options, standalone or using sidekiq.