Skip to content
Kuraib Ali

What is Web Bot Auth, and how do AI agents prove who they are?

A user-agent string can be copied by anyone. Web Bot Auth lets bots and AI agents sign each request, and Google has started testing it with its own agents.

By Kuraib AliPublished Updated 10 min read

A hand holding an open passport full of entry stamps, representing a bot identity checked against a trusted record
Photo: Unsplash

Web Bot Auth is a draft protocol that lets bots and AI agents sign their HTTP requests with a private key. A website, or the content delivery network (CDN) in front of it, checks each signature. It uses a public key the operator publishes at a fixed address. Google is testing it with AI agents on its own infrastructure, and an Internet Engineering Task Force (IETF) working group is writing the standard. It adds a check that a copied user-agent string cannot pass.

Why bot identity needs a new check

Most bots announce themselves with a user-agent string, such as Googlebot or Google-Agent. Any script can send the same string. That is why operators publish ways to confirm the claim.

Google's page "Verify requests from Google crawlers and fetchers" (last updated 20 March 2026) describes the two established methods. For one-off checks, run a reverse DNS (domain name system) lookup on the IP address and confirm the hostname ends in googlebot.com, google.com or googleusercontent.com. Then run a forward DNS lookup to confirm it points back to the same IP. For large-scale checks, match the IP against Google's published IP lists.

The AI crawlers guide covers these checks in detail. These checks tie identity to network addresses.

What Web Bot Auth is

Google's documentation describes Web Bot Auth as "an experimental cryptographic protocol used to authenticate requests sent by bots." It adds that "Web Bot Auth allows agents to cryptographically sign their requests" instead of relying only on self-reported headers and IP addresses.

The protocol builds on an existing standard. RFC 9421 (Request for Comments 9421), "HTTP Message Signatures", was published in February 2024 as an IETF Proposed Standard. It defines how to sign parts of an HTTP message and how a receiver verifies them.

Cloudflare's developer documentation explains the two drafts layered on top. A directory draft lets a bot operator "share their public keys". A protocol draft defines how those keys are used to attach the bot's identity to its requests.

Who is writing the standard

The IETF chartered a Web Bot Auth working group. Its charter covers "cryptographically authenticating automated clients and providing additional information about their operators to Web sites." The charter adds that its products "are intended for use by sites that primarily serve human users."

Until the working group finishes, the details may change. Google says the draft "may change over time."

How a signed request works

The flow has three parts. The table below follows the steps in Google's and Cloudflare's documentation.

StepWho does itWhat happens
Publish keysThe bot operatorHosts a public key set at /.well-known/http-message-signatures-directory on its own domain
Sign the requestThe bot or agentAdds Signature, Signature-Input and Signature-Agent headers to each signed request
VerifyThe website or its CDNFetches the operator's key set, then checks the signature using RFC 9421

Cloudflare's guide calls the /.well-known/http-message-signatures-directory path a requirement for bots it verifies. It supports the Ed25519 key algorithm. Operators register the key directory through the Cloudflare dashboard. After Cloudflare approves the registration, the bot can send verified requests.

A ring of old brass and iron keys on an open book, representing the public key set an agent operator publishes for websites to check

What Google is testing with Google-Agent

Google's page "Authenticate requests with Web Bot Auth (experimental)" was last updated on 13 October 2026. It says Google is "testing the protocol with some AI agents hosted on Google infrastructure."

The agent Google names on the page is Google-Agent. Google's list of user-triggered fetchers says Google-Agent "is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Its requests come from IP ranges published in user-triggered-agents.json.

When Google-Agent signs a request, Google's page gives the details a verifier needs:

  • the signed requests are authenticated as https://agent.bot.goog
  • the Signature-Agent header is set to g="https://agent.bot.goog"
  • the public keys are published at https://agent.bot.goog/.well-known/http-message-signatures-directory
  • the signature is checked with the Signature and Signature-Input headers labelled g, following RFC 9421

Google also gives two handling rules. Cache the key set according to its Cache-Control header, and delete cached keys that disappear from the file, since "they might be expired or revoked." Validate the signature's own expiry window separately from that cache period.

Why Google calls it experimental

Google lists what experimental means in practice. "Not all Google user agents are using Web Bot Auth." "Google is not yet signing every request of agents using the protocol." Google recommends that site owners keep "relying on IP addresses, reverse DNS, and user-agent strings" during the rollout.

Google also explains the slow pace. It says user-agent and IP-based checks are "the de facto standard, with decades of systems, policies, and best practices built around it." For latency-sensitive requests, Google allows a site to respond first and validate the signature afterwards, within its expiry window.

Signed agents on Cloudflare

Cloudflare uses Web Bot Auth as one route into its verified bots programme. Its verified bots page lists three ways a bot can identify itself. They are a signed Web Bot Auth request, a published IP range paired with a fixed user-agent and reverse DNS. The bot must also obey robots.txt and crawl directives.

Cloudflare's page records one change in plain terms: "Signed agents are now Verified." As of 1 July 2026, Cloudflare tracks the difference in a metadata field with two values.

Access typeCloudflare's description
DirectOne operator runs it, typically on infrastructure it controls
IntermediaryA service that many different end users can drive

Cloudflare points out a trust issue with intermediaries. The operator supplies the software, while a different end user starts each action. In Cloudflare's words, this "introduces transitive trust". In other words, you may trust the operator without trusting every person using it.

Bronze wax seals pressed onto three blush envelopes, representing a cryptographic signature that proves which agent sent a request

How Web Bot Auth compares with reverse DNS and IP lists

The three verification methods answer the same question in different ways. The table below sets them side by side. The limits for reverse DNS and IP lists are my own reading. The Web Bot Auth row follows Google's page.

MethodWhat the site checksMain limit
Reverse DNSThe hostname behind the IP, confirmed with a forward lookupNeeds a DNS lookup per new IP, and ties identity to the operator's network
Published IP listsWhether the IP sits in the operator's published rangesLists change, so the site must refresh them
Web Bot AuthA signature checked against the operator's public key setDraft standard, and Google signs only some requests today

The first two methods identify the network a request came from. Web Bot Auth identifies the signing key. My reading is that this matters most for agents on shared cloud infrastructure. There, an IP address says little about who sent a request.

Google's guidance treats the methods as layers. A site can accept a valid signature, then fall back to the IP and DNS checks for requests that carry none. My reading is that this keeps real but unsigned traffic flowing while the signed share grows.

Header formats can differ between providers

In my view, a draft standard leaves room for implementations to differ. Cloudflare's Web Bot Auth guide lists the cases in which it will fail to verify a message. One of them is a message that "uses the dictionary form from later drafts", with sig2="https://signature-agent.test" given as the example.

Google-Agent's Signature-Agent header, as Google documents it, is g="https://agent.bot.goog". My reading is that this is a labelled value of the same kind. A provider following the older form may not verify Google's signed requests yet. Google's own fallback advice covers this gap, since an unverified request can still pass the IP and DNS checks.

The same Cloudflare guide lists other failure cases to know about. A Signature-Agent value must be an https:// address in double quotes. The header must also be included in the components listed in Signature-Input. A site that verifies signatures itself should test against these cases before relying on the result.

What a signature does and does not tell you

A valid signature answers one question. It shows whether the request came from the operator that holds the key. It does not say how that agent behaves.

Google's documentation makes the gap clear for its own agents. User-triggered fetchers, the group that includes Google-Agent, "generally ignore robots.txt rules" because a user asked for the fetch. A signed Google-Agent request may therefore reach a page that robots.txt disallows for crawlers. The robots.txt and AI crawlers guide explains which bots follow which rules.

My reading is that Web Bot Auth is most useful for allow decisions. Bot protection tools can trust a signed agent without a guessing game over IP ranges. It is weaker as a blocking tool today, because Google says not every request is signed.

What site owners should do now

In my view, most of the work sits with your CDN or security provider. Google's page says: "Major bot-detection services, CDNs, and WAFs support Web Bot Auth." Steps 1 to 3 follow Google's guidance. The rest are my own recommendations.

  1. Ask your provider. Confirm whether it verifies Web Bot Auth signatures, whether it recognises Google-Agent and which header formats it accepts.
  2. Keep the old checks. Leave reverse DNS and IP list verification in place, as Google recommends.
  3. Avoid blocking unsigned Google-Agent traffic. Fall back to IP-based checks, since Google signs only some requests.
  4. Log the headers. Record Signature-Agent in your server logs so you can see which agents sign their traffic.
  5. Check that headers survive. Make sure a proxy or load balancer does not strip the Signature, Signature-Input or Signature-Agent headers before verification.
  6. Review your rules for agents. Decide what user-triggered agents may do on your site, since robots.txt may not govern them.

In my view, the log step is the most useful one this month. It costs little and shows whether signed agents reach your site at all. None of this changes rankings or AI citations on its own. It decides which agents your site can recognise and let in. For how this fits with the rest of AI search, see the AI search hub.

Related: do AI crawlers read your site, robots.txt and AI crawlers, how Google handles HTTP status codes.

Frequently asked questions

What is Web Bot Auth in simple terms?

Web Bot Auth is a draft protocol that lets a bot or AI agent attach a cryptographic signature to each HTTP request. A website can check that signature against a public key the operator publishes. A copied user-agent string is then no longer enough to pass as that bot.

Does Google sign all of its crawler requests with Web Bot Auth?

No. Google's documentation calls its implementation experimental. It says not all Google user agents use Web Bot Auth and that Google is not yet signing every request of the agents that do. Google advises keeping IP address, reverse DNS and user-agent checks in place.

Should I block Google-Agent requests that are not signed?

Google's guidance points the other way. It says to fall back to IP-based verification because not all requests are signed. An unsigned request that comes from Google's published IP ranges can still be real.

Does a signed request mean the agent follows robots.txt?

No. A signature proves who sent the request, and the rules it follows are a separate question. Google's documentation says its user-triggered fetchers, a group that includes Google-Agent, generally ignore robots.txt rules because a user requested the fetch.

Do I need to change anything on my site today?

In my view, most sites do not. Google says major bot-detection services, content delivery networks (CDNs) and web application firewalls support Web Bot Auth, so the check usually happens at that layer. Ask your provider whether it verifies signed agents and how it treats unsigned ones.