Is Traffic Claiming to Be GPTBot Genuine? How Web Bot Auth Works

Published Updated 27 min read
Is Traffic Claiming to Be GPTBot Genuine? How Web Bot Auth Works

A GPTBot name alone does not prove who sent a request. Learn how to check official IP ranges and signatures, how Web Bot Auth differs from RFC 9421, and what remained in draft as of September 2026. Includes a checklist for your web agency and guidance on recording unverified traffic.

A request claiming to be GPTBot is not necessarily genuine. The name alone cannot tell you. You can check its source information or the digital signature attached to the request. Web Bot Auth sets rules for using signatures to identify a bot’s operator. As of September 2026, it was still a draft [2][7].

If your access logs show that “GPTBot traffic tripled last week,” someone may ask whether it really came from OpenAI. Anyone can send the name Mozilla/5.0 ... GPTBot/1.4. Classifying a visit and confirming that it came from a particular company are different tasks.

Check a bot’s claimed identity. Name: self-reported. IP: compare the connection source. Signature: verify if supported. Even when the sender is identified, decide action permissions separately.

There are three types of checks: the reported name, the source address, and the signature. No single check tells you everything about the visitor. This article explains what you can establish, based on specifications and company guidance available as of September 7, 2026. Even if you do not manage the settings yourself, it can help you decide what to ask your web agency to do.

1. Check the name, source, and signature

The name is self-reported, the IP identifies the connection source, and the signature lets you check a match with a key. A UA is the name reported during a visit. An IP address is a number that identifies the connection source. A signature is verifiable information attached to a message using a particular key.

Layer What to check What it can prove Where it falls short
1. UA (User-Agent) A string in a request header, such as GPTBot/1.4 Only that the visitor claims that name Anyone can send the same string
2. IP range IP ranges in a company’s published JSON files, or reverse DNS That the request came from that company’s network Shared cloud IPs, reassigned IPs, and delays in updating public lists
3. Cryptographic signature The signature attached to the HTTP message and a public-key directory That the signature is valid for the corresponding key. The publisher, expiry, and other conditions must also be checked Nothing can be verified unless the sender signs requests. The specification is unfinished

You cannot establish authenticity from the name alone. Comparing the source with a published IP list helps, but you should also check when the list was updated and how the connection reached your site. The Web Bot Auth draft also explains that IP information alone can make it difficult to identify an operator [7].

Summary of the source: IP ranges alone may not be enough to identify the operator. [7]

An IP address helps you investigate where a request came from. An address match does not establish who the user is or which actions they authorized. Save the list you checked and the date and time of the check so you can review the result later.

In the cloud, several services may use the same provider’s infrastructure, and assigned IP addresses can change. The party using an address now may not be the one that used it before. Do not identify a particular AI company simply because an address falls within a large cloud provider’s range.

With signatures, the sender marks the message using a privately held key. The recipient checks it with a published verification key. Web Bot Auth sets rules for applying this method to bots. Cloudflare explains that the aim is to identify operators through request signatures, including for bots that interact with sites much like people do [10].

Summary of the source: The proposal aims to identify bot operators through signatures, even when their bots access sites in human-like ways. [10]

However, a valid signature does not mean the sender’s behavior is desirable. Site terms and action permissions still apply to identified senders. The presence of a signature is not a reason to grant access to every page.

2. RFC 9421: the foundation for signatures

RFC 9421 provides the basic mechanism for signing and verifying selected parts of a message. Its formal title is “HTTP Message Signatures,” and it was published as a Proposed Standard in February 2024 [1]. Web Bot Auth uses this existing mechanism for bots.

HTTP is the mechanism browsers and programs use to exchange page information. RFC 9421 defines how to create signatures or authentication codes for parts of those messages, and how recipients verify them [1].

Summary of the source: It defines how to create and verify signatures or authentication codes for selected parts of a message. [1]

You do not need to calculate signatures yourself in routine site operations. Ask your developers whether they use compatible systems or components and verify messages according to the specification. Adding a custom header with a similar name does not count as RFC 9421 verification.

Headers are information attached to a message. Because an intermediary delivery service may rewrite some of that information, the specification lets senders choose what the signature covers. It does not automatically protect the entire message. You need to check which fields were signed.

Item What RFC 9421 defines
Defined headers Signature-Input (what was signed) / Signature (the signature value) / Accept-Signature (a request for a signature)
Signature parameters created / expires / keyid / nonce / tag
Derived components that can be signed @authority / @target-uri, among others: the hostname or URL itself
Registered algorithms RSA-PSS / RSA v1.5 / HMAC-SHA256 / ECDSA P-256 and P-384 / Ed25519 / JWS-based algorithms
Replay protection No built-in enforcement mechanism. The design uses created, expires, and nonce to reduce the risk (§7.2.2)

You also need protection against attacks that resend a signed message unchanged. This is called a replay attack. RFC 9421 alone does not automatically stop every repeated request. Web Bot Auth specifies parameters such as expires and recommends short validity periods. Beyond checking the signature calculation, you must check expiry and reuse.

What to check with a signature. Whose key: confirm the party is trusted. Signed components: identify which information is being verified. Verification result: check that the content matches.

3. The status of Web Bot Auth

The working group was established on October 23, 2025, and its first working group draft was published on September 1, 2026. Two scheduled deadlines were missed, and discussions about the approach and its operation continue. Distinguish the publication of a draft from the completion of a formal standard.

Date Event Source
2025-03 Side meeting at IETF 122 in Bangkok [2]
2025-07-21 BoF at IETF 123 in Madrid: a meeting to consider forming a working group [2]
2025-09-26 Mark Nottingham submitted a second BoF request, which was approved. It noted “strong and urgent interest in solutions in this area” [2]
2025-10-23 The Web Bot Auth (webbotauth) working group was formally established. Its charter was approved. Chairs: David Schinazi and Rifaat Shekh-Yusef [2][3]
2025-11-04 IETF 124. Strong objections to the charter’s direction (Section 5) [4]
2026-04-13 Interim meeting. Votes on individual use cases (Section 5) [5]
2026-04-30 Charter deadline 1: submit a Standards Track document on authentication technology and communicating bot information to the IESG. Missed [2]
2026-07-22 IETF 126 in Vienna, with 75 participants. Vote on whether to use HTTP Message Signatures as the starting point: Yes 22 / No 6 / No opinion 5 [6]
2026-08-18 A chair posted a call to adopt the Meunier proposal to the mailing list. More than 35 replies [8]
2026-08-31 Charter deadline 2: submit an operational BCP document to the IESG. Missed [2]
2026-09-01 First working group document, draft-ietf-webbotauth-httpsig-protocol-00, posted. 44 pages, Standards Track. Authors: Thibault Meunier (Cloudflare) and Sandor Major (Google) [7]

August 31, 2026, was the target date for submitting an operational document, not the completion date for the standard. After nearly a year of working group activity, when the draft appeared on September 1, the group had not yet published a single RFC. Do not confuse this work with the already published foundation, RFC 9421.

A charter is a document that defines the purpose and scope of the work. The Web Bot Auth charter aims to identify operators and communicate additional information [2].

Summary of the source: The goal is to use cryptography to identify programs that access sites automatically and convey operator information to those sites. [2]

This gives sites evidence for deciding whose program accessed them. It is not a mechanism for checking a person’s identity document or proving that they agreed to buy a product. When using the term “identity verification,” specify whose identity is being checked to avoid confusion.

The scope includes search crawling, web-page archiving, broken-link checking, fetching content for AI training, and AI acting on a person’s behalf [2]. User login and API authentication for services are outside its scope. An API is an interface through which programs use each other’s functions. Even if the bot’s operator is known, permission to use that interface must be checked separately.

4. How signatures and keys are exchanged

The September 1 draft defines where verification keys are published and what information accompanies a request. These are specific rules for using RFC 9421 signatures to identify bots [7]. Developers should also check which draft version they are using.

Summary of the source: It defines information identifying the publisher, the format of the key list, and a common publication location. [7]

Signature-Agent indicates where the keys are published. JWKS is the format for the key list. A well-known URI is the common publication location. You can follow the process by thinking of these terms as “publisher information,” “key list,” and “designated location.”

The detailed settings are listed below. Rather than entering all of them manually, site operators should check which version their delivery service or verification components support.

Item Requirements in working group document -00
Signature-Agent header A Dictionary-type Structured Header. Its value is the HTTPS URL where the keys are published
Key directory location /.well-known/http-message-signatures-directory
Media type application/http-message-signatures-directory+json
Required parameters Both created and expires are required. keyid is the JWK’s SHA-256 thumbprint, encoded as base64url. tag is web-bot-auth
Validity period A period of no more than 24 hours is recommended
Signed components At least one of @authority or @target-uri, plus the signature-agent field
Algorithms Limited to those registered with IANA. Ed25519 is an example, not the only option

The recipient retrieves the keys from the specified publisher and verifies the request signature. But a sender pointing to a key it created does not prove it is a well-known AI company. Check that the publisher URL belongs to the intended provider, that the retrieved key is correct, and that the signature and expiry meet the requirements.

“A key exists,” “signature text is present,” and “verification succeeded” are three different states.

The draft recommends the following validity period [7].

Summary of the source: A validity period of no more than 24 hours is recommended. [7]

That recommended upper limit does not mean the same request can be accepted repeatedly during those 24 hours. Check how the system prevents replay and duplicate processing for each type of action. Signature expiry and duplicate-order prevention, for example, may require separate settings.

Cloudflare’s guidance says one minute may be enough in some cases [11]. But shorter is not always better: verification must account for transmission delays and differences between clocks. Follow the implementation’s guidance, and record expired signatures separately from invalid ones.

The two drafts used through March 2026, draft-meunier-web-bot-auth-architecture and draft-meunier-http-message-signatures-directory, have been superseded by the merged document [7].

When reading explanations from the first half of 2026, do not assume an old document name refers to the latest specification. Share materials showing the specification version and update date with your developers.

Checks after receiving a signature. Receive the information: obtain the signature and the information it covers. Check the key: confirm a trusted publisher. Verify the signature: check the content and conditions.

5. Identity verification is not permission to act

A central concern is that knowing an operator’s identity does not mean it will behave well. At IETF 124 and subsequent meetings, participants discussed which problems the mechanism should solve [4][6]. Signature technology alone does not determine access policy.

At IETF 124 on November 4, 2025, Eric Rescorla, Martin Thomson, and Brian Campbell said the problem needed to be clarified first. Alissa Cooper also questioned using identity as a substitute for judging behavior [4].

Summary of the source: Knowing an identity is not a substitute for good behavior. [4]

For example, even a verified bot run by a large provider needs attention if excessive requests overload your site. Conversely, a small research program is not necessarily malicious just because it does not support signatures. Keep identity checks separate from observations of actual behavior.

The minutes also record concerns that favoring providers able to support signatures could make access harder for smaller bots. Co-chair David Schinazi said, in substance, that another exploratory meeting should have taken place before the working group was formed [4]. Do not reduce this discussion to a conflict between particular companies.

At the interim meeting on April 13, 2026, participants voted on individual use cases [5]. The results varied by use case.

Use case Yes No
Control access for individual bots 17 5
Track a bot even when its IP address changes 14 0
Distinguish bots that share an IP address 11 0
Align with robots.txt 3 9

Checks for changing or shared IP addresses drew support, while the robots.txt use case drew more opposition. However, a meeting vote is not approval of a final standard. Read the question alongside the vote counts.

An alternative anonymous bot authentication proposal comes from Rescorla and Richard Barnes. draft-rescorla-anonymous-webbotauth-01 is dated July 19, 2026 [9]. It aims to show that a bot is authorized without revealing which bot it is. Its goal differs from approaches that reveal the operator’s name.

At IETF 126 on July 22, 2026, the vote on using HTTP Message Signatures as the starting point was Yes 22, No 6, and No opinion 5 [6]. Among those voting yes or no, roughly 20% opposed it. The record also notes that adopting operational recommendations would require a small change to the group’s scope.

Do not conclude that the work has failed simply because there is opposition. Check what is being debated.

For site operators, the practical point is that signatures do not complete an access policy. You still need to decide who may do what, how to treat parties without support, and how to stop problematic behavior.

Identifying a sender and granting permission are separate. Identity verification: who sent the request. Action permission: what that party may do. The site checks permission for each action.

6. Services with confirmed support

The review confirmed official implementations from Cloudflare and Vercel. It found no materials that established support from Akamai or Fastly. Google’s involvement in the specification must be distinguished from its own use of signatures. Unconfirmed does not mean unsupported.

Provider What was confirmed What was not confirmed
Cloudflare Offers Web Bot Auth as one verification method for Verified bots; documentation updated July 1, 2026. Supports Ed25519 only. Merged Signed Agents into Verified bots on July 1, 2026. Offers BotBase, a bot directory for Enterprise customers, with operator features dated August 28, 2026 —
Vercel Announced Web Bot Auth support for bot verification on August 12, 2025. Documentation updated August 11, 2026 lists three methods: IP, reverse DNS, and cryptographic verification. WAF rules can use the Signature-Agent header as a condition —
Google Working group document co-author Sandor Major is affiliated with Google. Proposals have been submitted for an IP publication format, jafar, and crawler best practices An official announcement that Google’s own crawlers or Google-Agent sign requests
Akamai — Documentation required login, so its contents could not be checked. No relevant post was found on the official blog
Fastly — No official materials were found

A CDN is a service that delivers site content. Even services in the same category can differ in supported signature methods and eligibility conditions. Do not assume that other delivery services offer what was confirmed for Cloudflare and Vercel. Ask your provider how it verifies requests and how to read the records.

Cloudflare treats bots whose operators and stated purposes it has checked as Verified bots [12].

Summary of the source: These are bots and agents whose operators and stated purposes Cloudflare has checked. [12]

When you see “verified,” check what was actually verified. Methods may include signatures, IP addresses, or domain names. Do not interpret the label as meaning that every action is safe or that the user personally approved it.

Cloudflare lists three verification methods: Web Bot Auth, a stable UA with a public IP list, and reverse DNS [12]. Reverse DNS looks up the domain name associated with an IP address. It also distinguishes Direct operations, with a single operator, from Intermediary services used by multiple users.

Categories such as Search, Agent, Training, and Transact indicate purposes such as search, acting on someone’s behalf, training, and transactions.

The August 28, 2025 Signed Agents announcement lists ChatGPT agent, Goose, Browserbase, and Anchor Browser as initial partners [13]. A distinguishing feature is that it treats these as services acting on people’s instructions, not just bot company names.

Summary of the source: It targets agents that act on user instructions. [13]

However, identifying a service operator does not identify the user behind it or establish what that user authorized. Sites handling reservations or purchases should keep their normal login and transaction checks. Do not treat a signature as blanket user consent.

BotBase for Operators, dated August 28, 2026, describes a feature that automates verification during registration [14].

Summary of the source: It automates checks of IP addresses, domain names, and signatures. [14]

This covers automatically retrieving IP lists, matching domain names, and verifying signatures. It can reduce manual checking, but it does not automatically determine each site’s permission policy. Use the results to decide which actions and traffic volumes to accept.

Vercel describes signatures as a way to verify senders without depending on the source network [15].

Summary of the source: It says signatures allow verification without relying solely on the source network. [15]

The advantage is that you can verify a match with the appropriate key even if the IP changes. But you still need to check the key’s publisher and expiry correctly. A rule that checks for the presence of Signature-Agent is not the same as a rule based on successful signature verification [16].

How to read support information. Supported features: what is supported. Conditions of use: where it can be used. Review date: when the guidance applies.

7. Published names, IP ranges, and signature information

Companies publish bot names and source information, but the full range of products that sign requests remains unclear. The table below lists information checked in September 2026 for six companies: OpenAI, Anthropic, Google, Perplexity, Apple, and Meta [17][19][20][21][22][23][24]. Publishing a key and signing individual requests are different things.

Provider Main published UAs Published IP ranges Signatures (Web Bot Auth)
OpenAI GPTBot / ChatGPT-User / OAI-SearchBot / OAI-AdsBot gptbot.json on openai.com: 21 prefixes; chatgpt-user.json: 184 prefixes, updated September 4, 2026; searchbot.json: 35 prefixes; adsbot.json One Ed25519 key published at the well-known location on chatgpt.com, with signature_agent: https://chatgpt.com and purpose: ai. However, OpenAI’s bots page does not mention signatures
Anthropic ClaudeBot (training) / Claude-User (user-triggered) / Claude-SearchBot (search quality) claude.com/crawling/bots.json: 26 IPv4 prefixes, updated August 18, 2026 No mention. The well-known location returns 404
Google Google-Agent (an agent acting on user instructions) / Google-GeminiNotebook. Google-Extended is a robots.txt token and has no separate UA Five JSON files. user-triggered-agents.json was created March 3, 2026 and contains four prefixes No official announcement; involved in developing the specification
Perplexity PerplexityBot / Perplexity-User perplexitybot.json on perplexity.ai: eight prefixes; perplexity-user.json: four prefixes No mention. The well-known location returns 404
Apple Applebot. Applebot-Extended controls training use through robots.txt applebot.json: 33 prefixes, updated July 31, 2026; or reverse DNS matching *.applebot.apple.com No mention
Meta facebookexternalhit / meta-webindexer / meta-externalads / meta-externalagent (training and indexing) / meta-externalfetcher (user-triggered) The current page does not describe IP verification No mention

Keep three points in mind when using this table: which requests are signed, how robots.txt is handled, and when the lists were updated. The numbers are a record from the review date, not fixed configuration values.

First, OpenAI publishes a key list [18]. Cloudflare describes signature use by ChatGPT agent, and Vercel describes it for ChatGPT operator [13][15]. But the existence of published keys does not prove that every OpenAI product signs its requests.

OpenAI’s guidance provided only limited confirmation of which products sign which requests. Nor can we conclude that OpenAI is the only provider able to use signatures.

Second, robots.txt may be handled differently when a page is read at a person’s request. OpenAI says its rules may not apply to ChatGPT-User in some cases [17].

Summary of the source: Because these actions follow user requests, robots.txt may not apply. [17]

robots.txt is a file that tells page-reading programs what is allowed or disallowed. It does not physically block access. Companies may treat user-requested retrieval differently from automated crawling.

Google-Agent, Perplexity-User, and Meta-ExternalFetcher also have descriptions of retrieval based on user requests [20][22][24]. Anthropic, by contrast, describes robots.txt controls for its programs, including Claude-User [19].

Do not conclude that “no agents can be stopped.” For more detail, see AI crawler types and settings.

Third, public IP lists change. The ChatGPT-User record showed 184 prefixes in the September 4, 2026 update [17]. A prefix represents a range of IP addresses. It does not mean 184 devices.

If you check against a year-old list, it may be missing genuine sources used today.

8. Common mistakes with dates and specification names

Submission targets, merged drafts, and the range of products signing requests are especially easy to confuse. Checking these three points helps prevent incorrect conclusions based on older explanations.

  1. August 31, 2026, was not the planned completion date for the standard. It was the charter’s target date for submitting an operational BCP document to the IESG, and it was missed [2]. Far from being a completed standard, the first working group document appeared the next day, September 1 [7].
  2. Do not treat the architecture and directory drafts as the latest documents. Both have expired and have been merged into draft-ietf-webbotauth-httpsig-protocol-00 [7].
  3. Do not assume all OpenAI requests are signed. Primary sources establish only that chatgpt.com publishes a key directory and that Cloudflare and Vercel say ChatGPT agent / operator signs requests [13][15][18]. There is no documentation showing that GPTBot signs requests.

Related authorization mechanisms should also be considered separately. Cloudflare’s “The Agent Access Model,” dated August 5, 2026, concerns AI permissions to access services and tools. It references RFC 8693, RFC 9449, and MCP [25].

MCP’s August 22 direction also mentions DPoP and Workload Identity Federation [26]. These names concern ways to pass permissions securely or verify identities between systems. They are not the same specification as Web Bot Auth, which helps sites identify visiting bots.

Watch for similar specification names. Formal name: identify the document. Document status: still under discussion or finalized. Update date: avoid confusing it with older guidance.

9. Bot-checking procedure and record sheet

Check the reported name, official source information, and signature information in that order. Record how far you got through the three levels. Logging whether a signature is present is not verification. If your delivery service already offers verification, also confirm how to interpret its results.

9-1. Checklist

Step Action What it establishes
1 Extract the strings in the Section 7 table from access-log UAs, such as GPTBot / ChatGPT-User / ClaudeBot / Claude-User / Google-Agent / PerplexityBot / Perplexity-User / Applebot / meta-externalagent A list of the bots visitors claim to be
2 Download each company’s public JSON files: openai.com/gptbot.json, claude.com/crawling/bots.json, Google’s user-triggered-agents.json, the JSON files on perplexity.ai, and Apple’s applebot.json. Check whether the source IP is included Whether the claimed name matches the source
3 Count non-matches separately as “unverified.” Do not immediately block them, because JSON updates can lag The number not confirmed through official lists
4 Log whether the request contains all three headers: Signature-Agent / Signature-Input / Signature The share of requests carrying signature information, which is separate from successful verification
5 Use the latest JSON for matching, even if you report only once a month. Match your update frequency to the provider’s Whether the list is current

Do not immediately label an IP as spoofed just because it is absent from the list. Published information can change, as with the September 4 ChatGPT-User update [17]. Ask the person responsible whether the recorded source IP belongs to an intermediary delivery service and whether the matching file is current. Keep an “unconfirmed” category.

9-2. Record sheet: copy and use

Month Claimed identity (UA) Requests IP match IP non-match Signature headers present Notes
2026-09 GPTBot
2026-09 ChatGPT-User
2026-09 Claude-User
2026-09 Google-Agent
2026-09 Perplexity-User

Three months of records will help you track the volume of unconfirmed traffic. However, the IP non-match count is not a spoofing count. Cloudflare’s July 1, 2026 report included a figure of 50% non-human traffic [28]. Another figure showed that user-triggered AI retrieval increased more than 15-fold in 2025 [27].

These are not spoofing rates. Be equally clear about what your own records count.

9-3. If you verify signatures yourself

Cloudflare publishes signing and verification components called cloudflare/web-bot-auth. Updates for Rust and npm were still continuing as of September 5, 2026. The draft is at version -00, so further changes are possible.

Before building your own implementation, check whether your existing delivery service can do the job. Consider whether you can maintain it through specification changes and key updates, and handle verification failures.

10. What AgentSignal’s visitor classifications tell you

AgentSignal’s human, AI agent, and bot classifications are estimates based on visit information. They are not signature-based identity verification. The current classification uses reported names, known bot information, some IP rules, and browser environment information.

For visits recorded by the tag, AgentSignal first checks rules such as known-name matches. If those do not match, it uses IP and environment information. It checks some Google-related ranges, but it does not compare every visit with the latest IP lists from all companies in the table. It also has a process for reviewing unknown names.

Even if the classification says OpenAI, distinguish that from proof, based on signature verification, that the request came from OpenAI.

You can review recorded visits in the interface as follows. Some retrievals may not be recorded, and some actions may not be fully classified.

  • Dashboard (ダッシュボード): Shows session counts for the past seven days and the “Agent ratio (エージェント比率).” Classification uses the information that could be collected; it does not reconstruct all Direct visits.
  • Web Recordings (Web 録画): The Type (種別) column shows Human (人間), AI Agent (AI エージェント), or bot. A referrer such as chatgpt.com provides a clue about the route taken, but the referrer alone cannot establish whether the visitor was human or AI. AI agents are recorded like humans. Bots are not recorded and do not consume the recording quota.
  • Filters (フィルタ): Filter by Type (種別) to display an AI-agent-only or bot-only list with one click.

Signature verification is not presented as an available feature of the current classification system. Checking for Signature-Agent, retrieving keys, verifying signatures, and checking expiry each require implementation. A draft update to -01 or -02 does not automatically complete support.

Users should review their delivery service’s verification results separately from AgentSignal’s visit classifications.

When reporting that “GPTBot traffic tripled,” specify whether you counted claimed names, IP-matched requests, or requests with verified signatures. Filling in three columns will not tell you everything, but it will let you distinguish confirmed from unconfirmed traffic. For tag setup, see Install the tracking tag. For checking on-screen actions, see What is WebMCP?.

Reading name-based classifications alongside records of actual actions makes it easier to decide where to investigate further.

Information to include when requesting an investigation

Simply saying “a suspicious bot visited” leaves the investigator unsure which requests to examine. Provide the date and time, affected page, recorded name, and source IP. If a delivery or protection service sits in front of your site, say whether the record comes from that service or from the site itself.

The visible IP can differ depending on where the same request was logged.

“Can you verify the sender using the official method?” is a clearer request than “Can we trust this name?” Ask the investigator to return the official IP list used, the date of the check, and, if a signature was present, whether verification succeeded. For list non-matches or verification failures, separate known causes from cases still under investigation. Do not report unexplained cases as fakes.

When discussing a response, present both the identification result and the actual problem. Even a genuine bot may need rate limits if a burst of requests slows the site. An unidentified sender is not necessarily malicious. Specify which pages and actions are affected, and how serious the impact is. This makes it easier to avoid an overly broad response, such as shutting down access across the site.

Classification and identity verification are separate. Classification: estimate the visitor type from records. Identity verification: check official information or signatures. Action permission: decide separately what to allow.

Conclusion

Claiming a name and proving an identity are different things. Check the source and signatures, and state how far verification went when you report the result.

  • Bot identification has three layers: UA, a self-reported name; public IP ranges, which indicate the source; and cryptographic signatures, which relate to the key holder. The working group document itself says that relying only on IP blocks can produce a confusing picture.
  • RFC 9421, the signature foundation, was completed in February 2024. For the bot-specific Web Bot Auth rules, the IETF working group was formed on October 23, 2025, and its first document appeared only on September 1, 2026. Both charter deadlines, April 30 and August 31, were missed. It has published zero RFCs.
  • Discussion continues over the purpose and approach. At IETF 124, objections included “identity is not a proxy for reputation” and concerns about creating a “fast lane.” The IETF 126 direction vote was Yes 22 / No 6, while an alternative anonymous-authentication proposal continues in parallel.
  • Implementations were confirmed for Cloudflare, which supports Ed25519 only, and Vercel. OpenAI publishes a key directory on chatgpt.com, but there is no official OpenAI documentation identifying which products sign requests. No signature descriptions were found for the other five companies.
  • Companies handle robots.txt differently, even for user-requested retrieval. Do not include Claude-User in a blanket category of agents that ignore it.
  • What you can do today is a three-stage check—UA → public IP JSON → presence of signature headers—and monthly recordkeeping. AgentSignal’s three classifications are estimates based on names, some IP information, and environment information. They do not indicate verified signatures, as of September 7, 2026.

FAQ

Q. Is Web Bot Auth already a standard?
There is no RFC for Web Bot Auth itself yet. The working group was established on October 23, 2025, and its first working group document was posted on September 1, 2026. April 30 and August 31 were target dates for submitting documents, not deadlines for completing the standard.
Q. How does RFC 9421 differ from Web Bot Auth?
RFC 9421 defines how to sign and verify web communications. It was published in February 2024. Web Bot Auth sets rules for using that method to identify bots. Topics under consideration include where to publish keys and how to handle expiry.
Q. How can I check whether traffic claiming to be GPTBot is genuine?
First, compare the source IP with the ranges OpenAI publishes in files such as gptbot.json. The name alone is not enough. If a signature is present, it must also be verified with a trusted party’s key. The public materials reviewed did not establish which OpenAI products sign requests.
Q. Can robots.txt stop AI agents?
robots.txt does not forcibly block access. Bots handle it differently. It may not apply to ChatGPT-User or Perplexity-User in some cases, while Claude-User is described as respecting it. Private pages need restrictions such as login requirements.
Q. Can services other than Cloudflare verify bot signatures?
Official public materials confirmed implementations from Cloudflare and Vercel. Cloudflare supports Ed25519 signatures. The materials reviewed did not establish support from Akamai or Fastly. Check with your service provider.
Q. How does AgentSignal identify AI agents?
It classifies visitors as humans, AI agents, or bots using rules for names sent by browsers and other software, registered classification rules, and some IP rules. This is not identity authentication that checks every company’s full IP ranges. It is not presented as an RFC 9421 signature-verification feature.

Sources

  1. [1] RFC 9421: HTTP Message Signatures (IETF / RFC Editor) — accessed 2026-09
  2. [2] Web Bot Auth (webbotauth) Working Group — About / Charter (IETF Datatracker) — accessed 2026-09
  3. [3] WG Action: Formed Web Bot Auth (webbotauth) (IETF Announce (mail archive)) — accessed 2026-09
  4. [4] Minutes IETF 124 webbotauth (2025-11-04) (IETF Datatracker) — accessed 2026-09
  5. [5] Minutes interim-2026-webbotauth-02 (2026-04-13) (IETF Datatracker) — accessed 2026-09
  6. [6] Minutes IETF 126 webbotauth (2026-07-22) (IETF Datatracker) — accessed 2026-09
  7. [7] draft-ietf-webbotauth-httpsig-protocol-00: HTTP Message Signatures for automated traffic (IETF Datatracker) — accessed 2026-09
  8. [8] web-bot-auth mailing list archive (IETF Mail Archive) — accessed 2026-09
  9. [9] draft-rescorla-anonymous-webbotauth-01 (IETF Datatracker) — accessed 2026-09
  10. [10] Forget IPs: using cryptography to verify bot and agent traffic (Cloudflare Blog) — accessed 2026-09
  11. [11] Web Bot Auth — Verified bots documentation (Cloudflare Docs) — accessed 2026-09
  12. [12] Verified bots policy (Cloudflare Docs) — accessed 2026-09
  13. [13] The age of agents: cryptographically recognizing agent traffic (Cloudflare Blog) — accessed 2026-09
  14. [14] BotBase for Operators (Cloudflare Blog) — accessed 2026-09
  15. [15] Vercel's bot verification now supports Web Bot Auth (Vercel Changelog) — accessed 2026-09
  16. [16] Bot management (Vercel Docs) — accessed 2026-09
  17. [17] Overview of OpenAI crawlers (OpenAI) — accessed 2026-09
  18. [18] http-message-signatures-directory (chatgpt.com) (OpenAI) — accessed 2026-09
  19. [19] Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic Support) — accessed 2026-09
  20. [20] Google user-triggered fetchers (Google-Agent) (Google Search Central) — accessed 2026-09
  21. [21] Verifying Googlebot and other Google crawlers (Google Search Central) — accessed 2026-09
  22. [22] Perplexity crawlers (PerplexityBot / Perplexity-User) (Perplexity Docs) — accessed 2026-09
  23. [23] About Applebot (Apple Support) — accessed 2026-09
  24. [24] Meta Web Crawlers (Meta for Developers) — accessed 2026-09
  25. [25] The Agent Access Model (Cloudflare Blog) — accessed 2026-09
  26. [26] The New MCP Roadmap (Model Context Protocol Blog) — accessed 2026-09
  27. [27] Cloudflare Radar 2025 Year in Review (Cloudflare Blog) — accessed 2026-09
  28. [28] Content Independence Day, one year on (Cloudflare Blog) — accessed 2026-09

About the author

Shogo Mizushima

CEO of kairos Inc. / AgentSignal Developer

Develops AgentSignal, a tool for measuring AI crawler visits and AI-referred traffic, and diagnosing AIO readiness. Writes about measurement and practical improvements for AI search using observed data.

Related articles