krawler v0.3.1 Release Notes

Release Date: 2017-02-02 // over 5 years ago
    • Created 1:1 mapping between threads and the number of queues used to serve URLs to visit. URLs have an
      affinity for a particular queue based on their domain. All URLs from that domain will end up in the same
      ๐ŸŽ queue. This improves parallel crawl performance by reducing the frequency that the politeness delay
      effects requests. For crawls bound to fewer domains than queues, the excess queues are not used.
    • ๐Ÿ›  Many bug fixes including fix that eliminates accidental over-crawling.