Skip to content
siterank.info
Sign in Start free

AI crawlers

PerplexityBot and Perplexity-User Explained

Perplexity runs two agents, and states that one of them generally ignores robots.txt. Here is what each does and how to configure for it.

By ankitkumarvig@gmail.com Published September 9, 2026 Last reviewed September 9, 2026 Sources verified September 9, 2026 6 min read

Perplexity documents two web agents, and neither of them is a training crawler. That already makes its roster different from OpenAI's or Anthropic's, and it changes what a robots.txt rule is actually deciding. One further statement on Perplexity's page deserves to be read before any rule is written: its user-triggered fetcher generally ignores robots.txt. Everything below comes from Perplexity's own crawler documentation, read on 9 September 2026.

The two agents

Agent Perplexity's stated purpose robots.txt Published IPs
PerplexityBot "Designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Respected; Perplexity recommends allowing it https://www.perplexity.com/perplexitybot.json
Perplexity-User "Supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response." "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." https://www.perplexity.com/perplexity-user.json

Evidence tier for this table: OFFICIAL PROVIDER GUIDANCE.

Perplexity states that each setting works independently and that changes may take up to 24 hours to be reflected in its systems. Both agents carry an explicit disclaimer that they are not used to collect content for training AI foundation models — Perplexity-User is documented as "not used for web crawling or to collect content for training AI foundation models" as well.

Full user-agent strings, as published:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

PerplexityBot: a search crawler, and only that

PerplexityBot is the search crawler that builds the index Perplexity's answers draw on. Perplexity's recommendation for sites that want to appear in its results is twofold: allow PerplexityBot in robots.txt, and permit requests from the published IP ranges. The second half is not decoration. A permissive robots.txt is inert if a WAF, CDN rule or security plugin is returning 403 to the request before it reaches WordPress, and that mismatch is the single most common cause of "we allow it but it never crawls us".

The absence of a training crawler in Perplexity's roster is the practical difference from OpenAI and Anthropic. There is no Perplexity equivalent of GPTBot or ClaudeBot to make a licensing decision about. Disallowing PerplexityBot is therefore purely a visibility decision: you are choosing not to be surfaced and linked in Perplexity's results. That may still be the right call — bandwidth, paywalled content, or a licensing position on answer engines generally — but it should be made knowing that is all it buys.

Perplexity-User: read this before writing a rule

Perplexity-User fetches a page because a person asked a question that needed it, and Perplexity says it may include a link to the page in the response. Its documented robots.txt behaviour is the sentence to internalise: since a user requested the fetch, this fetcher generally ignores robots.txt rules.

That is a clear statement, and it is different from Anthropic's position on its own user-triggered agent, which Anthropic documents as controllable through robots.txt, and from OpenAI's more conditional "robots.txt rules may not apply" for ChatGPT-User. Three companies, one functional category, three different documented behaviours. Any advice that treats "user-triggered AI fetchers" as a single configurable group is wrong somewhere.

Two consequences:

  1. A Disallow for Perplexity-User is not a control. Writing one is not harmful, and it does record your preference legibly for anyone auditing the file, but do not report it to a client as a block. If a page must not be fetched on a user's instruction, the enforcement layers are authentication, a paywall, or an edge rule — not robots.txt. RFC 9309 has always described an advisory protocol.
  2. Blocking it does not remove you from Perplexity. Presence in Perplexity's results is governed by PerplexityBot. The two are independent.

Worked robots.txt

Available to be surfaced and linked in Perplexity results:

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap_index.xml

Not available in Perplexity results, with a stated preference against user-directed fetching that Perplexity documents it generally will not honour:

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

Section-level restriction, which is usually a better instrument than a site-wide rule:

User-agent: PerplexityBot
Disallow: /members/
Disallow: /pricing-internal/
Allow: /

As with every operator, rules are per host. example.com/robots.txt does not govern docs.example.com.

Firewall and CDN configuration

Perplexity is unusual in documenting WAF setup directly, which tells you how often the edge is the real blocker. Its guidance, paraphrased:

  • Cloudflare: create a custom WAF rule under Security → WAF that matches User-Agent containing PerplexityBot or Perplexity-User and a source IP in the published ranges, with the action set to Allow so the request bypasses other security rules.
  • AWS WAF: create IP sets from the published endpoints, create string-match conditions on the User-Agent header for both tokens, combine them into allow rules, and associate those with your Web ACL at a higher priority than blocking rules.
  • Both: fetch the IP ranges from the official JSON endpoints rather than hard-coding them, and automate periodic refresh — Perplexity warns the addresses are updated regularly and that the JSON endpoints are the source of truth.

Perplexity's stated best practice is to combine user-agent matching with IP verification rather than relying on either alone. That is the correct posture generally: a user-agent header is trivially forged, and matching on the string alone is an open invitation to any scraper that copies it. Verifying by IP is the only check that means anything.

What not to assume

  • Do not assume Perplexity-User respects a Disallow. Perplexity says it generally does not.
  • Do not assume blocking Perplexity protects content from training. Perplexity documents neither agent as collecting content for foundation model training, so there is nothing there to block. If training use is your concern, the relevant tokens belong to other operators.
  • Do not assume allowing PerplexityBot produces citations. It makes you eligible to be surfaced and linked. Perplexity documents no mechanism for how a source is selected for a given answer, and neither does any other provider. Crawled is not retrieved; retrieved is not cited.
  • Do not judge a robots.txt change immediately. Perplexity documents up to 24 hours for changes to be reflected.
  • Do not trust a log entry on its user-agent string. Check it against the JSON list for that agent.

Evidence classification

Statement Tier
Each agent's purpose, user-agent string and IP list OFFICIAL PROVIDER GUIDANCE
PerplexityBot respects robots.txt; Perplexity recommends allowing it plus its IP ranges OFFICIAL PROVIDER GUIDANCE
Perplexity-User generally ignores robots.txt OFFICIAL PROVIDER GUIDANCE
Neither agent collects content for foundation model training OFFICIAL PROVIDER GUIDANCE
Combine user-agent matching with IP verification in a WAF OFFICIAL PROVIDER GUIDANCE
Allowing PerplexityBot increases citations in Perplexity answers Not supported at any tier

Key takeaways

  • Perplexity publishes two agents and no training crawler, so a robots.txt decision here is about visibility, not licensing.
  • PerplexityBot governs whether you can be surfaced and linked in Perplexity results; allowing it in robots.txt is only half the job if your edge blocks it.
  • Perplexity-User generally ignores robots.txt by documented policy. A Disallow records a preference; it does not enforce one.
  • Verify by the published JSON IP lists, combined with user-agent matching, and automate the refresh.
  • Changes can take up to 24 hours to take effect.

Official sources & further reading