AI crawlers are eating your bandwidth — how to allow the good ones and block the rest

The user-agent string was never a security control

Your logs are full of GPTBot, ClaudeBot, Google-Extended, and a dozen other crawler names. Some of them are the real thing, indexing or training on your public content. Many of them are not — a scraper spoofing a well-known name because it knows most sites wave it through on the string alone. A user-agent is a self-declared label. Anyone can type it. Blocking or allowing traffic on that basis is theatre, and it always was.

The real question is not what does this client call itself but can this client prove it is who it claims to be. That is the question Smart WAF answers at the edge, before the request ever reaches your origin.

Verified bots prove themselves cryptographically

Well-behaved crawlers increasingly sign their requests. Instead of trusting a name, the edge verifies a cryptographic HTTP message signature the crawler attaches — the top rung of our challenge ladder, Web Bot Auth. A signature that checks out against a published key is a client that genuinely controls the identity it claims. A missing or invalid signature is a client pretending. The difference is provable, not guessed.

This flips the default from deny everything automated to allow what you can verify. A verified search or AI crawler you actually want indexing your content passes cleanly. A scraper wearing its name does not, because it cannot produce the signature. You are no longer choosing between blocking useful bots and opening the door to freeloaders.

You decide the policy — the edge enforces it

Not every site wants the same thing. A documentation portal might welcome AI indexing for the reach. A subscription publisher might want its paywalled archive left alone. A SaaS dashboard has no reason to serve any crawler at all. So the policy is yours to set per domain, and the edge enforces it consistently on every request:

• Allow verified crawlers, challenge the rest. Good bots that sign in pass; unsigned automation gets escalated up the challenge ladder — proof-of-work, then a CAPTCHA — so a stubborn scraper pays a real cost per request.
• Rate-shape the greedy ones. Even a legitimate crawler can hammer an origin. Pair crawler policy with rate limiting so no single client, verified or not, drowns your backend.
• Score the ambiguous middle. Clients that neither sign in nor obviously misbehave get an ML entity score from their behavior across requests, so a slow-and-quiet scraper is still caught by pattern rather than by name.

What we don’t claim

We do not maintain a magic list of “all the good bots” and promise it is complete — such a list is stale the day it ships. We verify identity where a crawler offers proof, and we score behavior where it does not. We also do not promise to block every scraper on earth; a determined adversary running a residential proxy farm and solving challenges by hand can get through anything, and anyone who tells you otherwise is selling. What we give you is a control that is honest about its basis: cryptographic proof over spoofable strings, your policy over our guesswork, enforced identically on every request through the edge.

All of this runs inline, in the request path, inside our added-latency ≤5 ms design budget — a verified-bot check is a signature verification, not a round-trip to somewhere else.

Getting the policy you actually want

The right crawler policy depends on your business, not on a default someone picked for you. If you are drowning in scraper traffic, or you want AI indexing on your terms instead of theirs, tell us your domain and we will set the policy up with you and watch the first traffic together.