SuiteCommerceChecker

How much we read

An audit has a page budget fixed before it starts, and the crawler stops at it. We keep at most 2 page 2 requests and 4 API 4 requests in flight to one storefront at a time, and space them by at least 750 ms and 250 ms respectively. A Crawl-delay in your robots.txt raises that spacing, and we honor yours over ours whenever yours is longer.

The anonymous pre-check is one request for your home page and nothing else. It never sends seodebug, preview or seonojscache, so it cannot make your account regenerate anything.

Letting us through a firewall

Allowlist on the user-agent substring SuiteCommerceChecker/. We do not publish a fixed IP range: the crawler runs on shared infrastructure and the addresses change, so an address allowlist would break without warning. If your WAF needs one, ask and we will tell you what the current egress looks like at that moment, with the caveat attached.

What robots.txt does, and the one case where it does not stop us

A Disallow matching our user agent stops a partner audit and the anonymous pre-check outright, with no exception. Nobody consented on your behalf, so your file is the whole answer, and the report says the audit was refused rather than reporting a clean site.

On an audit requested by a verified owner of that domain it does not stop us, and you should know that before you rely on the file. We fetch page paths anyway, and four public read-only endpoints under an /api/ disallow — /api/cacheable/items, /api/items, /api/navigation/v1/ and /api/cms/v2/. The reasoning is that the person asking is the person the file belongs to, and a file written to keep search engines out of a faceted catalog is rarely an instruction to the owner about their own audit. Every override is recorded, and the Disallow comes back in the report as a finding, because it is more often a mistake than an intention. The allowlist is closed: any other /api/ path is refused even for an owner.

A sandbox — a test copy of a storefront that its owner verified and linked to their production domain — is audited exactly the same way: an owner’s audit of it overrides robots.txt like any owner audit, and records the override. A sandbox is usually kept out of search engines on purpose, so its report lists the robots, sitemap and indexing checks that are about the live storefront as not checked on a sandbox, rather than as defects. If the sandbox answers 401 — it asks for a password — the audit stops at that first request, reports the site as password-protected and refunds its credits. We do not crawl behind a password.

One path is honored absolutely, for every kind including an owner: the SSP application path we read from your live site. Crawling it renders the application as a bot and forces a regeneration for no informational gain.

So if you want one control that stops us in every case — including an audit somebody on your own team requested — block the user agent at your WAF, or ask us to stop and choose “everything”.

Turning us off

• Block the user agent at your WAF. That stops every request of every kind immediately and needs nothing from us. A Disallow in robots.txt also works immediately against partner audits and the pre-check, but not against an audit requested by a verified owner of the domain — the section above says why, and what to use instead if that is the case you care about.
• Verify the domain yourself and switch off partner audits — for every site under it when you verified by an address at the domain or verified the bare domain itself, which is then the default, or for that hostname alone. It covers the sandboxes you link to it as well, and an audit of an open site that redirects to one you closed stops at the redirect. A verified owner’s opt-out blocks every agency account from requesting your storefront — refused before any credit is spent, with no appeal path through our support and no plan that overrides it. It stays in force if your verification later lapses; changing it needs a current one. We notify you of each attempt, because “somebody tried” is something you are entitled to know; that notice names no agency, and you can decline those notices without declining anything else. The domain page does name them: every agency account that audited a site under your domain, when and how often, and its refused attempts — names shown when you verified with a DNS record or a tag in the page head and are not an agency yourself, counts otherwise. We also tell you the first time each agency audits your site. You can allow one agency while refusing the rest, or refuse one while allowing the rest. Refusing stops new audits; the reports an agency already made stay with it.
• Ask us to stop, and prove the site is yours with a DNS record, a link sent to an address at the domain, or two lines in your robots.txt. The moment you ask, agency audits and our anonymous check of the site pause for two days while you publish the proof. Once it is proven, what you chose takes effect for good: agency audits only, or everything — every account and every kind of crawl, matched on the hostname (or every hostname under the domain) rather than on an account we have, so a storefront we have never audited is refused just as a customer’s is. An audit already running stops at the next stage boundary.
• We ask for proof because stopping everything also stops the site’s owner auditing it, and we will not do that on the word of somebody who might be a competitor or a supplier. If an account has verified the site, we tell it that the request was made and proven, and how to reach us if it was not them.
• Anything else — a request we should look at by hand, a domain someone else holds, a question — contact us, or write to [email protected]. No account needed.

What we keep and for how long is in the privacy notice. We never publish an audit of a storefront, ours or anybody else’s.