AI crawler access SOP | GEO Playbook

Network disclosure. All ten sites in this network are published by Bay Area AEO. They cover answer engine optimization in the Bay Area from ten angles and link to one another so readers can move between related topics.

Owner: Site admin
Cadence: Quarterly, and after any hosting or security change
Time box: One hour
Output: robots.txt copy, fetch tests, validation log

Why access comes before everything on the site

An AI crawler is the software an AI company uses to fetch web pages, such as GPTBot, ClaudeBot or PerplexityBot. If the crawler is refused, the service pages, FAQs and markup built in the other procedures cannot be read or quoted, however good they are. Bay Area AEO checks this as part of its work: if a site blocks AI crawlers or is very hard to read, the agency says so and recommends the smallest fix.

Blocks happen by accident more often than by choice. A security plugin, a hosting default or a one-click setting at the CDN can turn crawlers away without anyone at the business knowing.

Step one: robots.txt

Open the site's robots.txt at the root of the domain. Look for any Disallow: / under a user agent an AI company uses, and for a blanket rule under User-agent: *. The file below allows the main AI crawlers explicitly; adapt it, keeping any rules the site needs for private areas.

User-agent: *
Allow: /

# OpenAI: search results, user-requested fetches, model training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
Allow: /

# Anthropic: search, user-requested fetches, model training
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
Allow: /

# Perplexity: index and user-requested fetches
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Google and Apple AI-use tokens, and Bing (which Copilot reads)
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bingbot
Allow: /

Sitemap: https://[canonical domain]/sitemap.xml

What each crawler does

The companies separate their crawlers by purpose. OpenAI's OAI-SearchBot gathers pages for ChatGPT's search answers, ChatGPT-User fetches a page when a user's conversation calls for it, and GPTBot collects training data. Anthropic and Perplexity split their crawlers the same way. Google-Extended is a separate control token for whether Google may use pages for its Gemini models; Google's AI Overviews are built from the regular Googlebot crawl, so Googlebot access matters for them. Applebot-Extended is Apple's equivalent token, and Bingbot feeds the Bing index that Copilot reads. A business that wants to be recommended needs, at a minimum, the search and user-fetch crawlers allowed.

Step two: the firewall and CDN

robots.txt is a request; a firewall is a wall. Some CDNs and security services, Cloudflare among them, offer a setting that blocks AI crawlers, and on some services it may be switched on for new sites. Ask whoever manages hosting to check bot management settings, firewall rules and rate limits for the crawlers above. Note the answer and the date in the log.

Step three: rendering

Many AI crawlers read the HTML the server sends and do not run JavaScript. A page whose service text loads only after a script runs may look complete in a browser and nearly empty to a crawler. Use the browser's "view source" (not the inspector) on each service page and search for a sentence from the answer paragraph. If it is missing, the page needs server-side rendering or static output for its text.

Validation log: one row per check, kept with the date.
Check	Tool or method	Pass condition
robots.txt	Read the live file	No rule blocks the search or user-fetch crawlers
Firewall and CDN	Ask the host; check bot settings	AI crawlers not blocked or challenged
Rendering	View source on each service page	Answer paragraph present in raw HTML
Sitemap	Open sitemap.xml	Every service, FAQ and neighborhood page listed
Server logs	Search access logs for the crawler names	Recent successful fetches recorded

Procedure summary

Read robots.txt
Save a dated copy, then edit it to allow the crawlers above.
Ask the host
Get a written answer on bot blocking at the firewall or CDN.
View source
Check every service page for its answer paragraph in raw HTML.
Check the sitemap
Confirm all new pages are listed and the sitemap is named in robots.txt.
Read the logs
A week later, look for successful fetches by the crawler names.
Log the result
Fill the validation log and file the robots.txt copy.

Done when

robots.txt allows search and user-fetch crawlersHost confirms no AI crawler block at the firewall or CDNAnswer text present in raw HTML on every service pageSitemap complete and referencedCrawler fetches visible in the logs

The mistakes that turn up most

Four problems account for most blocked sites. The first is a leftover development rule: a Disallow: / under User-agent: * that was meant for a staging copy and went live with the launch. The second is a plugin that writes its own robots rules, so the file on the server is not the file anyone edited. The third is a CDN bot setting switched on during a traffic spike and never reviewed. The fourth is a site builder that renders text in the browser, which reads perfectly to a person and nearly blank to a crawler.

Each one has a quick test in the validation log, which is why the log runs every quarter and after any hosting change, even when nobody remembers changing anything.

Pages that should stay private

Allowing AI crawlers does not mean opening everything. Keep rules that block admin areas, checkout steps, customer portals and internal search results. Those pages hold nothing a customer's question would need, and keeping crawlers out of them leaves the crawl for the service pages, FAQs and neighborhood pages that answer questions.