Our crawler
Crawling is disabled by default and requires per-merchant opt-in. Most data comes from feeds merchants supply directly.
Identification
Where crawling is enabled, requests identify themselves as AgentProductGraphBot/0.1 with a link to this page and a contact address.
What we honour
- robots.txt, per RFC 9309, including Allow/Disallow precedence, wildcards and Crawl-delay.
- Your sitemap, including the
Sitemap:directive in robots.txt. We prefer being told what you have over guessing from category pages, and we uselastmodso an unchanged page is not re-crawled. - Retry-After on 429 and 503 responses.
- Conditional requests using ETag and Last-Modified, so an unchanged page is not re-downloaded.
- One concurrent request per domain by default, with a configurable delay.
What we never do
- Bypass a CAPTCHA, a login, a paywall or any bot protection.
- Use stealth or anti-detection tooling, or rotate proxies to evade a block.
- Ignore robots.txt, or treat a 5xx response from it as permission.
- Follow a sitemap entry pointing at another site. A sitemap speaks for its own domain and nothing else.
- Continue after repeated 403 or 429 responses — the source pauses itself automatically.
Opting out
Contact us from an address on the domain through the developer support form, or add a Disallow rule for our user agent. We will also honour a request to remove data already collected.