How do AI crawlers discover new pages? Reading Cloudflare and OpenAI’s pilot results

Early results from a Cloudflare–OpenAI research pilot show how network signals can help AI crawlers discover pages they did not previously know about. The work aims to expand discovery while avoiding unnecessary fetching. Its effects on individual websites’ visits and business outcomes remain a subject for further research.[1]

Publishing a detailed manual does not immediately reveal whether ChatGPT has found that page. An AI crawler is a system that visits web pages to gather information for purposes such as AI search.

The announcement is easier to interpret by separating discovery, fetching, citation in an answer, and a person’s visit. This explanation is based on official material checked on October 16, 2026. It distinguishes the pilot’s findings, existing management features, and suggested checks for site operators.

The Cloudflare–OpenAI pilot to improve AI crawler discovery

The news is an October 15 report of early results from a research pilot launched in July. Cloudflare provides infrastructure for website delivery and traffic protection; OpenAI provides ChatGPT. The companies are testing whether signals available through Cloudflare’s network can help discover information for ChatGPT.[1]

The central question is how published information is found. This matters to pages such as manuals and specialist articles whose usefulness does not, by itself, establish that a search system has discovered them.

Start with the announcement’s timing, the participating population, and the meaning of the reported numbers. Without those distinctions, research progress can be mistaken for a feature available to every site today or for evidence of increased traffic.

Separate the July launch from the October 15 results

The research pilot began in July 2026, and the early results were announced on October 15. The announcement does not say a new feature became available to all websites on that day. It separates findings so far from work planned for the next phase.[1]

That timeline also changes how to describe the news internally. It supports reviewing the research population and future availability, rather than announcing that a newly available feature now needs to be configured.

When	What the announcement establishes
July 2026	Cloudflare and OpenAI launched the research pilot
October 15, 2026	Early results on discovery and related work were published
Next phase	Further testing of page selection, change signals, and outcomes after discovery

The last row has no confirmed launch date. A direction for further research is different from a dated availability commitment.

For anyone who read the original launch announcement, the new element is the report of results. Keeping the initial proposal and later observations separate makes it easier to identify what each subsequent announcement actually adds.

The participating cohort covered about 1% of Cloudflare’s network

The reported results come from a participating cohort representing roughly 1% of Cloudflare’s total network. That is not 1% of all websites worldwide, 1% of Japanese websites, or 1% of the pages on each site. The denominator must remain the one specified in the announcement.[1]

The report establishes that the research used a participating population. It does not provide the precise number of sites, a breakdown by industry or country, or confirmation that the cohort was randomly selected.

For a business publishing Japanese manuals, a relevant question would be whether comparable pages were included. A percentage of the overall network does not answer that question. This is a limit on applying the result to a particular business, rather than a reason to dismiss the research.

When sharing the figure, keep “Cloudflare’s participating network cohort” alongside “about 1%.” The number also does not establish that the remaining approximately 99% are ineligible for future use.

Read the results as findings from a defined research population, and check a particular site’s participation separately. This keeps progress in the pilot distinct from actions currently available to a site operator.

Hundreds of millions of indexed pages are not a site’s referral traffic

The companies report that OpenAI discovered and indexed hundreds of millions of web pages it had not previously known about. A search index is the system in which information about pages is registered for search. The figure describes discovery and indexing, rather than visitors or purchases.[1]

“New pages” does not necessarily mean every page was first published during the pilot. The comparison is with what OpenAI previously knew. A manual that was already publicly available could therefore be newly discovered in this sense.

Keep that unit intact when considering business implications. More discovered pages do not reveal how many were cited in a particular answer or how many people subsequently visited. Those questions require their own records.

Question	Information needed
How far did discovery expand?	The number of previously unknown pages
Was this website cited?	Links to its pages in answers
Did people visit?	Records of visits through those links or other referral paths

Use separate fields for these measures in a site report. The pilot-wide figure cannot be converted into a traffic growth rate for an individual business. Discovery is the starting point; what follows needs to be checked separately.

How Cloudflare signals help ChatGPT Search discover pages

Cloudflare supplies discovery signals, while OpenAI independently evaluates pages for relevance and usefulness. Cloudflare is not described as deciding the ranking of pages in ChatGPT Search. Finding information and deciding how to use it in search are separate responsibilities.[1]

For site operators, the news concerns whether a page’s existence is discovered, as well as what that page contains. An additional discovery route does not guarantee that every page will be selected.

The following sections use a manual page to distinguish knowing a URL from a person visiting it. This is an explanatory model, not a disclosure of OpenAI’s unpublished internal processing order or ranking formula.

Use network signals to find previously unknown URLs

Signals from Cloudflare’s network help OpenAI’s search crawler discover useful pages. The announcement gives examples such as model-specific manuals and local business opening hours. Its point is that pages can be useful even when few other sites link to them.[1]

A URL is a page’s address. Consider, as an illustrative example, a page explaining how to install a particular appliance model. Knowing the company’s homepage is not the same as knowing where that installation guidance is located.

Discovery signals matter here because the value of the information and awareness of its location are different issues. Detailed guidance can already be public, yet a system that does not know the page cannot proceed to assess its contents.

The announcement does not provide a complete field list for the signals exchanged or a submission form for site operators. An imagined set of fields should not be presented as a new required configuration.

A business can identify which public page it wants found and the question that page answers. For a manual, that means checking that the model and the scope of the instructions match. Understanding discovery provides a reason to review concrete public content without inventing an integration procedure.

OpenAI decides how pages are evaluated and ranked for search

OpenAI independently evaluates and ranks the pages. The pilot’s division of responsibilities keeps discovery through Cloudflare’s signals separate from the judgment that information is useful for search.[1]

Suppose a manual’s URL is discovered, but the question concerns a different appliance model. The manual’s existence does not make it the right answer. This illustrates why locating a page and assessing its fit to a question involve different judgments.

The announcement therefore does not support “use Cloudflare to improve rankings” as a guaranteed outcome. It describes responsibilities within a research pilot, not a product commitment to ranking gains for individual websites.

An internal evaluation can distinguish the confirmed capability—providing discovery signals—from the hoped-for outcome—having relevant pages discovered. Recording an expectation does not turn it into a measured effect.

The announcement also does not disclose detailed evaluation weights or a ranking formula. As an editorial practice, it is more useful to clarify a page’s subject, applicable conditions, and factual accuracy than to rewrite it around an unknown formula. That is a content-review recommendation, not an experimentally demonstrated ranking benefit.

Read discovery, fetching, citation, and visits as separate events

Discovery means finding a URL; fetching means requesting its contents. Citation means a page is identified as a source in an answer, while a visit means a person opens the website. Separating these events helps interpret the pilot’s report.[1]

Stage	Question in the manual example
Discovery	Was the URL of the relevant manual found?
Fetching	Could the instructions be received from that URL?
Citation	Was the page used as a source in an answer?
Visit	Did a person proceed to the manual page?

Success at one stage is not proof of success at the next. A fetch request in access records does not reveal where the page was used in an answer.

Likewise, seeing a link in one answer does not establish that the same page will appear for every later question. Recording the question and date makes the observation a defined reference point for later comparison.

In team discussions, replace an ambiguous statement such as “we appeared in AI” with the stage actually checked. A fetching problem calls for examining traffic; a problem after a visit calls for examining the reader’s experience. This announcement helps explain improvements at the discovery end of that chain.

Three examples of URLs that waste AI crawler requests

The announcement identifies nonexistent pages, URLs that all redirect to a homepage, and copies of the same content as sources of wasted fetching. More URLs or more requests do not necessarily mean more useful information. Recognizing these patterns before fetching is part of the pilot’s efficiency work.[1]

A request is a communication asking a website to return content. It is not a count of interested people reading the page. Automated requests should therefore not be treated directly as popularity or customer demand.

The three examples involve different checks: whether content exists, whether the intended information is reached, and whether there is distinct information. These distinctions make a large request count easier to interpret.

Requests for missing files are not proof of a popular page

Automated systems can repeatedly request files that do not exist. Cloudflare describes how that activity can make a nonexistent page appear interesting. Request counts alone can therefore obscure whether there is any content at the address.[1]

As an illustrative example, consider traffic to a misspelled manual URL. A request record does not establish that the correct manual exists there. Interpreting a high count as evidence of valuable information would assign value to a page that is absent.

A suggested operational check is to verify that a heavily requested URL contains the expected material. Open the page and, where necessary, ask its administrator to inspect the responses. This is a site-review practice, not a reconstruction of the pilot’s internal procedure.

Finding that a URL does not exist does not mean an article should immediately be created for it. If the source of the requests and the origin of the URL are unclear, demand from people has not been established.

Record the URL, observation period, returned response, and whether real content exists. This distinguishes an address receiving requests without delivering information from a page that actually contains material.

What happens when almost every URL redirects to the homepage

If nearly every requested address redirects to a homepage, fetching different URLs repeatedly leads back to the same entrance. The announcement identifies this as a source of unnecessary work. Reaching an accessible destination is different from reaching the requested information.[1]

In the manual example, a request for model-specific instructions repeatedly returns the company homepage. A company description can be readable without answering how to install that appliance.

To examine this difference, record the originally requested URL alongside the final displayed URL. A note that “the page opened” can otherwise omit the fact that the destination changed.

This does not make all redirects wasteful. The announcement’s example concerns a pattern of sending almost any address to the same homepage. It is not a basis for disabling appropriate redirects associated with a site move.

During a review, check whether the destination contains the relevant instructions. If it does not, give the administrator both addresses and the intended destination. The practical lesson is to identify which information should be reached, rather than simply removing redirects.

Copies of the same content are not additional information

Different URLs can lead to copies of the same content. Cloudflare includes this among the patterns relevant to wasted fetching. The number of page addresses and the variety of information should be read separately.[1]

Imagine the same instructions for the same appliance model appearing at two URLs. Two addresses do not automatically answer two different questions. Comparing the contents establishes whether anything has actually been added.

A useful review compares the subject, steps, conditions, and substantive changes, rather than titles alone. Similar titles can also describe genuinely different subjects, so a title match is not enough to classify pages as copies.

Compare	Difference to look for
Subject	Whether the same model or service is described
Body	Whether different steps or conditions are provided
Changes	Whether anything besides the displayed date has changed

This is a suggested way to review information on a site. It is not OpenAI’s disclosed duplicate-detection criteria or a specification of how it selects a preferred URL.

Even after identifying copies, do not remove pages before checking their purpose and existing references. First establishing whether there is distinct information prevents URL growth itself from being mistaken for a discovery achievement.

Using change signals to make AI crawler refetching more efficient

The freshness work tests signals that help identify when a page is worth fetching again. Fetching every page more frequently would repeatedly transfer unchanged content. The goal is to keep information current while reducing unnecessary work.[1]

Refetching means returning to retrieve a page that has already been read. A corrected manual and an unchanged manual offer different amounts of new information on another visit.

This is why total request counts alone are insufficient. Separate repeat visits to unchanged pages from checks of newly discovered pages, rather than treating load and discovery as a single measure.

Reduce repeated downloads of unchanged pages

Repeatedly downloading an unchanged page adds traffic load to the website. The pilot describes using change signals to determine whether another visit is worthwhile. Its purpose is not simply to reduce every kind of request.[1]

If a manual is identical to yesterday’s version, another download adds no new instructions. If an important precaution has been corrected, retaining the earlier information would be undesirable. Information about changes is relevant to distinguishing those situations.

A site’s own records also need more than a count of refetches to explain whether the work was necessary. Recording which page changed and when provides context when investigating later changes in traffic.

Creating that record does not submit data to the pilot. It is an operational suggestion for comparing a site’s updates and traffic, not a dedicated Cloudflare input format.

The announcement does not establish one ideal fetching interval. Rather than inventing an hourly or daily official recommendation, frame the research question as whether necessary fetching can be directed toward changed pages. That captures its purpose without supplying an unpublished schedule.

Discovering new pages can initially increase crawling

Finding new pages can initially increase crawling while their contents are evaluated and indexed. The official announcement explicitly discusses balancing that work with reductions in unnecessary fetching. It does not promise that request volume will always fall.[1]

For illustration, suppose several previously undiscovered manuals are found. Fetching them introduces new work. Even if redundant visits to existing pages decline, the total number of requests could still rise.

Operationally, separate two questions: how requests to previously known pages changed, and whether requests began reaching other public pages. This avoids deciding success or failure from the aggregate alone.

Observed change	What to check next
More requests to the same page	Whether its content or responses changed
Requests to additional pages	Which public information is now being requested
Fewer requests overall	Whether fetching of needed pages also stopped

The table is a suggested investigation after observing a change. If participation in the pilot has not been confirmed, a site’s traffic changes cannot be attributed to it. Keep the observation separate from the explanation of its cause.

Wait for further evidence on update speed and load reduction

The announcement does not provide a guaranteed update delay or a confirmed percentage reduction in load. Testing faster, more reliable change signals is part of the next phase. The direction is clear, but a service deadline or guaranteed performance value cannot yet be assigned.[1]

Consider a manual immediately after a correction. The time its body was updated and the time an AI answer used the new content are different records. Recording the first does not establish the second.

Even evidence of a refetch does not, by itself, confirm an answer has changed. Faster change signals, faster refetching, and faster updates in answers refer to different endpoints.

If later reports provide numbers, check the start and end of the measurement, the pages covered, and the comparison period. An average and a result under a particular condition support different expectations for an internal operating process.

For now, keep update dates separate from observed displays, and do not use this announcement to promise a fixed reflection time to customers. The research can be followed without filling in unpublished speed claims.

Separate discovery cooperation from permission for AI access

A signal about a page does not itself grant permission to access that page. The pilot’s explanation preserves the site owner’s choices. A clue that helps locate content and the right to read that content must be treated separately.[1]

Wanting a public manual to be found does not require giving the same access to members-only material. Decisions about which pages are available to whom should not be replaced by assumptions about discovery.

This section distinguishes access rights, training content, and existing allow/block controls. Their connection to AI does not make research participation, content provision, and access management a single operation.

Sharing a page signal does not automatically expand access rights

Information identifying a page is not permission to read it. The official announcement explicitly states that a page signal does not grant access. Here, a signal means information that helps find a page or assess its state.[1]

Consider an illustrative website with a public manual and separate supplementary material for members. Knowing the supplementary material’s URL is different from being authorized to read it. The pilot should not be understood as removing authentication when a URL becomes known.

Conversely, that statement is not an audit proving every private item on a site is protected. What a particular site actually delivers and where its access restrictions apply remain separate matters for the site’s operator to check.

A useful first distinction is between public information intended for discovery and information whose readership is restricted. If both appear on the same page, establish which portions are intended to be public before widening AI access.

The announcement is not a procedure for changing authentication. Its relevant point is that ownership decisions about public access remain in place while discovery is studied. Keeping this distinction makes it possible to discuss audience growth without confusing it with permission to view content.

Distinguish search metadata from page content for model training

The companies describe this research as using web metadata for search and discovery, rather than sharing page content for model training. Metadata is information about a page, as distinct from the page’s full contents.[1]

For example, information used to locate a manual and the complete text of that manual are different things. Describing the pilot as a mechanism for handing over full manuals for training would change the stated purpose of the research.

The statement applies to this research. It should not be extended into a blanket guarantee that content will never be used for training in any other access path or service.

In an internal review, identify which arrangement is under discussion. The purpose of information used in this pilot, a decision to allow a particular crawler, and content supplied under another agreement can have different conditions. Keeping them separate avoids combining unrelated commitments.

For background on search and training crawlers, see the guide to AI crawler roles. The conclusion supported here is the pilot’s stated use of metadata for search and discovery. There is no need to invent undisclosed fields or processing details.

AI Crawl Control is where existing allow/block choices can be reviewed

AI Crawl Control is Cloudflare’s feature for understanding and managing AI crawler access. Its documentation describes Allow and Block actions for individual crawlers. It is not presented as an enrollment button for this research pilot.[2]

To review settings, open the Cloudflare dashboard, select the relevant account and domain, and open AI Crawl Control. Appropriate account access is needed. Establish which domain and crawler are being reviewed before interpreting the settings.[2]

The documentation uses Security in its opening navigation instructions and Crawlers in the table explanation. Do not change an unrelated setting based on a single tab label. The relevant items are the crawler name, operator, and current Allow/Block action; reviewing them does not require changing the selection.

Any decision to change Allow or Block should follow the site’s access policy and the crawler’s purpose. Reading this announcement is not, by itself, a reason to allow every crawler.

The AI Crawl Control configuration guide provides more detail. This article explains official documentation; it is not a record of changing a live account or joining the pilot.

Harmful-page detection and business outcomes remain under investigation

The report also covers testing signals for harmful pages and plans to study business outcomes after discovery. The number of discovered pages is only one part of the work. Which pages should be excluded and what happens after discovery are separate research questions.[1]

Harmful-page detection concerns avoiding unsafe destinations. Visits and sales concern the results for the site publishing the information. Progress on one question does not establish results for the other.

Site operators can make a similar distinction by recording requests separately from referrals. The following sections distinguish what the pilot reports from what existing analytics can show.

Try to distinguish a compromised page from its entire domain

The pilot is testing signals that identify potentially harmful individual pages within otherwise legitimate sites. The announcement’s example is a fake login form planted by an attacker. It describes a page-level approach intended to avoid treating the entire site in the same way.[1]

A domain is the name identifying a site. Where it contains many pages, a problem on one page and the contents of the remaining pages are distinct matters. A valid manual and a problematic page elsewhere on the same site illustrate the difference.

The early results are not a guarantee that every harmful page will be detected. The announcement does not establish a definitive detection accuracy or miss rate, nor does it certify that a particular website has been inspected.

What the report describes	What needs separate evidence
Testing signals for problematic pages	Accuracy and applicable conditions
Distinguishing an individual page	Whether a particular site has a compromised page

This does not remove the need for a site administrator’s own security checks. Understanding a research direction and investigating an incident on a live site are different activities. Better discovery alone is not evidence that an entire site has been verified as safe.

Citations, visits, and sales after discovery still need measurement

The companies plan to continue investigating how better discovery translates into real business outcomes. The announcement does not establish a fixed increase in citations, visits, or sales for an individual site. The scale of discovered pages must not be converted into downstream performance.[1]

An operator may want to know whether the page reached people who actually needed it. For a manual, that means readers with questions about the relevant model. For a page intended to support purchasing, a purchase after a visit is a further event to measure.

When reading an outcome report, establish what counts as one result. An answer containing a source link, a page visit, and a completed application are different events.

Even an observed increase does not establish that the pilot alone caused it. Record other changes made in the same period, such as newly published pages or substantive updates, so the surrounding circumstances can be reviewed.

These distinctions make improvements easier to describe accurately; they do not dismiss the possibility of benefits. Future reports should be checked for the participating sites, measured behavior, period, and comparison. Business decisions need evidence corresponding to the outcome being claimed.

Record requests and referrals separately for the same period

AI Crawl Control provides request counts, response breakdowns, and referral information. Referrers indicate where traffic came from. The documentation places referral analytics behind a paid-plan condition, so an unavailable field should not be recorded as zero.[3]

As a suggested review, choose the relevant account and domain, then align the period in AI Crawl Control’s Metrics tab. Where useful, filter by crawler, operator, hostname, or path. A path is the part of a URL identifying the page’s location.[3]

Separate record	Meaning
All requests	Total requests, including blocked and error responses
Allowed requests	Requests receiving a successful 200–299 response
Status code distribution	The mix of success, redirects, errors, and other responses
Referral information	AI-service referral sources, subject to availability conditions

Many requests do not necessarily mean many successful content fetches. Keep referrals in another column to distinguish increased crawler activity from increased visits.

When using the documented Download CSV option, also record the applied period and filters.[3] This is a documentation-based review suggestion, not a measurement of the pilot’s effects in a live account.

What this announcement means for AI search optimization

A practical starting point is to identify a public page intended for discovery, examine its fetching records, and keep downstream outcomes separate. A site can consider reviewing its own content and current traffic without assuming participation in the pilot.

The announcement highlights that useful content also needs to be found and fetched when appropriate. It does not yet provide a complete adoption guide with confirmed participation conditions and business results.[1]

The final sections suggest a one-page review and identify the information that would change a decision when further results appear. The aim is to make the current state and unanswered questions concrete.

Choose one public page and record its current state

Start with one public page that answers a specific reader question. For a manual, identify the model and the task it explains. The following is a suggested site review, not a registration procedure for the pilot.

1. Open the published page and record its actual URL.
2. Write the question it answers in one sentence, then check that the body contains the answer.
3. Confirm the last substantive change using the site’s own update records.
4. If access records are available, choose a period and inspect requests and responses for that URL.

The documented analytics support filtering by path and period and examining requests and responses.[3] There is no need to fill in numbers from features the site does not use. Mark unavailable evidence as unverified.

If the page opens but does not contain the intended explanation, review the content first. If the content is present but no fetching record is found, record the period and scope checked rather than declaring the page undiscovered. The announcement alone cannot identify why a particular record is absent.

The immediate result is an account of one URL’s public content and the traffic evidence available. This is not proof of inclusion in an AI answer. Use the record to narrow the next question to investigate.

Participation for non-Cloudflare sites and Japanese sites is unconfirmed

The announcement does not establish general enrollment procedures, pricing, eligibility for Japanese websites, or an availability date. It also does not provide a path this article can recommend for sites not using Cloudflare. Unconfirmed signup routes and contract terms should not be supplied by assumption.[1]

That does not establish permanent ineligibility. The source is an early research report, rather than a comprehensive statement of general availability conditions.

An adoption review can use separate fields for the research population, the site’s eligibility, and required costs. Information in the first field does not fill the other two. Neither eligibility nor pricing can be calculated from the roughly 1% cohort figure.

Existing Cloudflare customers should also avoid treating the presence of AI Crawl Control as confirmation of pilot participation. Crawler management and research enrollment are separate matters to verify.[1][2]

When asked about participation internally, provide the official announcement and the date checked, and name the conditions that remain unannounced. Avoid asserting either immediate availability or exclusion of Japan. Keep a record that can be updated when eligibility information appears.

In the next announcement, check coverage and outcomes after discovery

Future reports should be examined separately for expanded availability and for evidence of downstream outcomes. The announcement identifies better page selection, further testing of change signals, and measurement after discovery as next-phase work.[1]

Item to watch	Question relevant to a site’s decision
Coverage and eligibility	Can the site’s domain and operating setup participate?
Change-signal testing	What start and end points define the measured delay?
Outcomes after discovery	Were citations, visits, or purchases counted?
Comparison conditions	Which period and population were used as the comparison?

Beyond a reported increase, ask how a site could verify the corresponding measure for itself. If the population or period is missing, treat it as information still needed rather than directly incorporating the number into an adoption forecast.

For day-to-day allow/block decisions, use the AI Crawl Control guide. For the distinction between search and training crawlers, start with the AI crawler introduction.

The confirmed development is progress in a joint experiment that adds signals for discovering pages. A site can clarify where its intended public information lives and record which stages—fetching, citation, and visits—have actually been checked. That provides a concrete basis for interpreting the next results.