binux/pyspider
pyspider, Source Code, на сайте с 16 декабря 2022, 12:47
Write script in Python
Powerful WebUI with script editor, task monitor, project manager and result viewer
MySQL, MongoDB, Redis, SQLite, Elasticsearch; PostgreSQL with SQLAlchemy as database backend
RabbitMQ, Redis and Kombu as message queue
Task priority, retry, periodical, recrawl by age, etc...
Distributed architecture, Crawl Javascript pages, Python 2.{6,7}, 3.{3,4,5,6} support, etc...