CrystalSoft/crawly
https://github.com/CrystalSoft/crawly, на сайте с 18 августа 2023, 23:58
A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules.