CrystalSoft/crawly
https://github.com/CrystalSoft/crawly, на сайте с August 18, 2023 23:58
A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules.