ChompShopperBot
If you run a recipe site and found this address in your logs, this page explains what our reader does and how to control it.
What it is
Chomp Shopper turns a recipe into a grocery list. When someone pastes a recipe link, adds a recipe from our search, or we pick a Recipe of the Week, our server downloads that one page to read its ingredients. It identifies itself as:
ChompShopperBot/1.0 (+https://chompshopper.com/bot)What it doesn't do
- It doesn't crawl. It reads the one page it was asked for and never follows links from it.
- It doesn't download images, scripts or styles, only the page itself.
- It doesn't republish your recipe. We keep the title, the ingredient list and the link, and every list links back to your page for the instructions.
How it behaves
- It reads your robots.txt (kept for a day) and skips any page it asks us not to read.
- It sends at most one request to your site every 2 seconds, or follows your Crawl-delay if it's longer (up to 10 seconds).
- If you answer 429 or 503, it waits for your Retry-After time, or backs off on its own, and tries at most twice more.
- After three 429 answers in a row, it leaves your site alone for 15 minutes.
- It keeps a page it has read for 30 minutes, so several people adding the same recipe cause one download.
- It doesn't try to get past bot checks or CAPTCHAs. If it gets one, it stops.
How to block or limit it
Add a group for it to your robots.txt. To block it everywhere:
User-agent: ChompShopperBot
Disallow: /To slow it down instead:
User-agent: ChompShopperBot
Crawl-delay: 5It also follows your rules for User-agent: * when there's no group with its name. If we can't load your robots.txt at all, we treat it as allowing reads and check again an hour later.
Allowing it
If your firewall or bot protection is blocking it and you're happy for your readers to make shopping lists from your recipes, you can allow the user agent above. Until then, our readers see a message that your site can't be read automatically and are asked to copy the ingredients from your page themselves.
Contact us
Questions, or something we got wrong? Email support@chompshopper.com or use our contact page. Recipe sites we read are listed on Supported Sites.