Crawls hammer sites with no per-request throttling
Scrapy has no per-request delay or robots Crawl-Delay support, so default crawls can hammer small sites.
Tier II · Emerging Problems that are still building up.
3 public reports · first reported 14 July 2014 · last reported 1 January 2025
What's going wrong
Scrapy users crawling unknown or small websites find that the default setup sends requests at full speed, even after the server answers with rate-limit errors.
AI-assisted: written with the help of AI from the reports. The idea report has the full summary.
In the idea report
- The full summary.
- The products the 3 reports are about, and how many came each month.
- Related problems.
No business has been found with this problem yet, so the report has no contacts. All sales are final — no refunds. Pricing.