Cobweb

web crawler that can use resque to cluster crawls to quickly crawl extremely large sites. Web crawler with very flexible crawling options, standalone or using sidekiq.