OpenAI Crawlers: GPTBot, OAI-SearchBot and ChatGPT-User

OpenAI publishes four web agents, each with a separate name, a separate purpose, a separate published IP list, and — critically — a separate robots.txt setting. The most expensive mistake a site owner makes here is treating them as one thing: a rule written for GPTBot says nothing about whether ChatGPT's search feature can surface your pages. Everything below is taken from OpenAI's own crawler documentation, read on 9 September 2026.

The four agents

Evidence tier for this table: OFFICIAL PROVIDER GUIDANCE.

OpenAI states the governing principle up front: it exposes OAI-SearchBot and GPTBot robots.txt tokens so that webmasters can manage how their sites work with AI, and "each setting is independent of the others". Its own worked example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.

OAI-SearchBot

This is the search crawler. It builds what ChatGPT's search features query.

The consequence OpenAI documents for disallowing it is precise, and the precision matters: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." So a Disallow here does not make you invisible inside ChatGPT — it removes you from the answers, not necessarily from every link surface. Do not tell a client that blocking OAI-SearchBot erases them from ChatGPT.

OpenAI's recommendation for sites that want to appear is to allow OAI-SearchBot in robots.txt and to allow requests from its published IP ranges — a reminder that a permissive robots.txt is worthless if an edge firewall is dropping the requests above it.

Example user-agent string, with a version number OpenAI says may change:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

A useful operational detail: when fetching robots.txt itself, OpenAI may add a robots.txt marker to the user-agent string, so that owners whose logs do not record paths can still tell a robots.txt request from a content request:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

GPTBot

GPTBot is the training crawler. OpenAI describes it as crawling content that may be used in training its generative AI foundation models, and states that disallowing it "indicates a site's content should not be used in training generative AI foundation models".

Read that as what it is: a signal of intent, honoured by OpenAI as a matter of policy. It applies to content crawled from that point onward. It is not a deletion request, it does not reach models already trained, and it has no documented effect on whether you appear in ChatGPT's search answers today. Whether to allow it is a content-licensing question, not a visibility question, and any tool that scores it as a defect is scoring an opinion.

Example user-agent string (version may change):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

ChatGPT-User

This is the user-triggered fetcher, and it behaves differently from the two above in ways that are easy to misreport.

OpenAI documents that when a user asks ChatGPT or a Custom GPT a question, it may visit a page with the ChatGPT-User agent, and that users may also reach external applications through GPT Actions. Three statements on that page are load-bearing:

1. "ChatGPT-User is not used for crawling the web in an automatic fashion." It fetches specific URLs on demand; it does not spider.
2. "Because these actions are initiated by a user, robots.txt rules may not apply." This is a documented conditional, not a promise in either direction. Do not claim ChatGPT-User obeys robots.txt, and do not claim it always ignores it.
3. "ChatGPT-User is not used to determine whether content may appear in Search." OpenAI directs owners to use OAI-SearchBot in robots.txt for search opt-outs and automatic crawling.

The practical reading: writing Disallow: / for ChatGPT-User is not a reliable control. If you need to stop a person's ChatGPT session from retrieving a page, the enforcement layer is authentication or an edge block, not robots.txt. And blocking it does nothing to your ChatGPT search presence — that is governed entirely by OAI-SearchBot.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OAI-AdsBot

The narrowest agent. OpenAI states it validates the safety of pages submitted as ads on ChatGPT, that it may use landing-page content to decide when the ad is most relevant to show, that it "only visits pages submitted as ads", and that the data it collects "is not used to train generative AI foundation models". If you do not run ChatGPT ads, this agent has no reason to visit you.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

What blocking each one does and does not affect

Two timing and efficiency notes from the same page. First, "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results — do not judge a change the same afternoon. Second, if a site allows both bots, OpenAI "may use the results from just one crawl for both use cases to avoid duplicative crawling". Allowing both does not double your crawl load.

Worked robots.txt

The most common intent — be citable in ChatGPT search, decline training use:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

The opposite intent, allowing training while staying out of ChatGPT's search answers:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Disallow: /

And a section-level restriction, which is usually a better instrument than a site-wide rule when the concern is a specific body of content:

User-agent: GPTBot
Disallow: /members/
Disallow: /courses/
Allow: /

Apply these to every host you care about. robots.txt is per host: example.com/robots.txt does not govern docs.example.com or shop.example.com.

Verifying the traffic is really OpenAI

A user-agent header is trivially forged, and scrapers borrowing GPTBot's string to look legitimate is a known pattern. The only verification OpenAI offers is the published IP lists — one JSON file per agent, linked in the table above. Match the requesting address against the file for the agent that the header claims to be; a GPTBot string from an address outside gptbot.json is not OpenAI.

If you run a WAF or CDN, check it before you check robots.txt. An Allow rule that a bot never gets to read because the edge returns 403 is the most common cause of "our robots.txt permits it but the crawler never arrives".

What none of this gets you

Allowing OAI-SearchBot makes you eligible to be retrieved. It does not make you cited, and it certainly does not make you recommended. OpenAI documents the crawl and the opt-out; it does not document how the model selects which retrieved passages to name in an answer, and no provider does. Crawled is not retrieved, retrieved is not cited.

That is also why crawler access cannot be an audit score. SiteRank AI reports the effective rule for each OpenAI agent, with OpenAI's own stated consequence beside it, and proposes changes as a diff you approve. It does not flag Disallow: GPTBot as a problem, because declining training use is a legitimate choice, not an error. If you want to know whether ChatGPT actually cites you, that is a measurement question, answered by observing responses to a fixed prompt set over time — see how LLM visibility monitoring works and our methodology.

Evidence classification

Key takeaways

• Four agents, four independent settings. OAI-SearchBot governs ChatGPT search presence; GPTBot governs training; ChatGPT-User is user-triggered; OAI-AdsBot only visits submitted ad pages.
• Disallowing OAI-SearchBot removes you from ChatGPT search answers but you may still appear as a navigational link.
• ChatGPT-User is documented as possibly not subject to robots.txt, and OpenAI says it plays no part in search inclusion.
• Robots.txt changes take about a day to take effect for search, and allowing both bots may still mean a single crawl.
• Verify by the per-agent IP JSON files, and check your edge before you blame your robots.txt.

Official sources & further reading

• Overview of OpenAI crawlers — OpenAI (all agent facts above; fetched 9 September 2026, HTTP 200)
• Published IP ranges — OpenAI: https://openai.com/searchbot.json, https://openai.com/gptbot.json, https://openai.com/chatgpt-user.json, https://openai.com/adsbot.json
• RFC 9309: Robots Exclusion Protocol — IETF
• AI features and your website — Google Search Central, for the contrasting position that no special optimisation is required for AI features

Related reading

• AI crawlers explained: search, training and user-triggered bots
• Anthropic crawlers: ClaudeBot, Claude-SearchBot and Claude-User
• robots.txt for AI crawlers
• Crawled is not cited
• SiteRank AI's AI crawler policy inspector