krawler v0.3.1 Release Notes
Release Date: 2017-02-02 // about 7 years ago-
- Created 1:1 mapping between threads and the number of queues used to serve URLs to visit. URLs have an
affinity for a particular queue based on their domain. All URLs from that domain will end up in the same
🐎 queue. This improves parallel crawl performance by reducing the frequency that the politeness delay
effects requests. For crawls bound to fewer domains than queues, the excess queues are not used. - 🛠 Many bug fixes including fix that eliminates accidental over-crawling.
- Created 1:1 mapping between threads and the number of queues used to serve URLs to visit. URLs have an