binux/pyspider

Write script in Python Powerful WebUI with script editor, task monitor, project manager and result viewer MySQL, MongoDB, Redis, SQLite, Elasticsearch; PostgreSQL with SQLAlchemy as database backend RabbitMQ, Redis and Kombu as message queue Task priority, retry, periodical, recrawl by age, etc... Distributed architecture, Crawl Javascript pages, Python 2.{6,7}, 3.{3,4,5,6} support, etc...