I Ran an AI Search Visibility Audit on My Own Site. Here's the Method.

This morning I sent a request to my own homepage with the user-agent set to GPTBot. The server answered with HTTP 200 and all 46,848 bytes of the page. Then I ran it again as OAI-SearchBot, ClaudeBot, and PerplexityBot. Four for four, full page every time.

That's the result I wanted. It's also a result most site owners have never checked, because "can AI search actually reach my site" sits in a blind spot between SEO tooling and CDN configuration. Search Console won't tell you. Your rank tracker won't tell you. Your firewall logs might, if you know what to look for.

So here's the full audit I ran on bree-sharp.com on July 17, 2026, written as a method you can repeat on your own site. Real numbers throughout, including the parts where the honest answer is "this file probably does nothing."

Step 1: Find out which robots.txt is actually serving

Not the one in your repo. The one a crawler receives. These can be different files.

Cloudflare offers a managed robots.txt that can block AI crawlers by default, and depending on how your zone was set up, it may be serving instead of yours. I run this site on Cloudflare and I deliberately disabled the managed file so my own robots.txt answers. If I hadn't, Cloudflare's version could have been quietly turning AI crawlers away while my repo copy said the opposite.

How to check: fetch https://yourdomain.com/robots.txt in a browser and read what actually comes back. Compare it line by line against the file you think you deployed. My robots.txt checker parses the live file and flags AI-crawler rules.

What I found: fully open.

User-agent: *
Allow: /

Sitemap: https://bree-sharp.com/sitemap.xml

No AI-bot disallows, no managed-file override.

What failure looks like: disallow lines for GPTBot or ClaudeBot that you never wrote. If you're on Cloudflare, check the AI Crawl Control settings before assuming your file is the one serving. And if you do find blocks, decide on purpose: GPTBot is OpenAI's training crawler, while OAI-SearchBot is the one feeding ChatGPT search. Blocking training while allowing search is a legitimate stance. Blocking both by accident is just a hole in your visibility.

Step 2: Test the firewall layer, not just the polite layer

robots.txt is a request. Your WAF is enforcement. One says what you ask crawlers to do; the other decides what they can do. A crawler can honor your Allow: / perfectly and still eat a 403 from a bot-blocking rule somebody added during a scraper panic two years ago.

How to check: send a real request with a spoofed AI user-agent and look at the status code and byte count.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" \
  https://yourdomain.com/

Repeat with the user-agent strings for OAI-SearchBot, ClaudeBot, and PerplexityBot.

What I found: all four returned 200 with the full 46,848-byte homepage. No challenge page, no truncated response.

One honest caveat. A spoofed user-agent from a residential IP doesn't perfectly replicate how a WAF treats verified bots, because Cloudflare and its peers verify real crawlers by IP range and signature. A rule scoped to verified bots won't fire on my test. But a user-agent-level block would have 403'd, and user-agent blocks are the most common way sites lock AI crawlers out without realizing it. This test catches exactly that.

What failure looks like: a 403, a 5xx, or a suspiciously small download. If your homepage is 46KB and the spoofed request pulls 2KB, that's a challenge interstitial, and an AI crawler hitting it gets nothing.

Step 3: Check the index each assistant actually reads

AI assistants don't crawl the live web for most queries. They query an index, and it usually isn't Google's. Claude's web search runs on Brave's index. ChatGPT search rides Bing. Which means a site can rank fine on Google and be functionally invisible inside an AI answer, because the model looked somewhere you never optimized for.

How to check: go to search.brave.com and Bing directly and search your brand name plus your niche. Not Google. The point is to see what the models see.

What I found: a Brave search for my name plus my niche returned 10 bree-sharp.com URLs, and when I asked Claude about me, it accurately synthesized my positioning, my services, and my pricing from those pages. That's the outcome you're after. Not just indexed, but coherent enough that the model gets you right.

What failure looks like: strong Google presence, thin or absent Brave presence. Brave crawls the open web, so the fix is standard indexability hygiene: crawlable pages, clean sitemap, no accidental noindex. My indexability checker covers the on-page half of that in one pass.

Step 4: Wire IndexNow for the Bing side

Since ChatGPT search reads Bing's index, freshness on Bing is freshness in ChatGPT. Bing supports IndexNow, a ping protocol that tells the index about new and changed URLs instead of waiting on crawl discovery. My IndexNow key is wired into my publish workflow and I ping on every deploy. In practice I see a 24 to 72 hour window from ping to updated Bing result, which sets the floor on how fast ChatGPT can reflect a change on my site.

What failure looks like: no IndexNow, so changes drift into Bing whenever the crawler wanders by. That can be weeks. For a time-sensitive page, weeks of lag in the index ChatGPT reads is a real cost.

Step 5: llms.txt — keep it cheap, expect nothing

I serve one at /llms.txt. Mine is a short machine-readable summary with entity links: who I am, what I do, canonical URLs for services and tools.

Now the honest adoption read, because this file has a hype problem. No major AI provider confirms consuming llms.txt in production. One 90-day server-log study counted 84 hits on the file out of 62,100 total AI bot visits, which is 0.1%. Google's John Mueller has confirmed no Google system reads it at all.

So why keep it? Because it cost me twenty minutes, it's a tidy entity summary, and honestly I'm a little stubborn about having my entity links stated somewhere machine-readable. That's the entire case. If a vendor pitches llms.txt as a citation lever, they're selling you a file that almost nothing reads. Walk.

Step 6: Score your content against what actually gets cited

Everything above is plumbing. Citations are what the plumbing feeds, and the industry research on who gets cited is blunt. These are published findings, not my data, but they're steering my decisions:

• Roughly 97% of Google AI Overview citations come from pages already in the top 20 organic results. AI visibility is not a parallel channel. It's mostly your existing rankings wearing a different interface.
• google.com is now the #2 most-cited domain in AI Mode, driven largely by Google Business Profile subpaths. For local queries, GBP completeness feeds AI answers directly. If you serve a local market, your Business Profile is now an AI-visibility surface, not a checkbox.
• Original data and proprietary research is the most-cited content type. A real audit with real numbers beats a rewritten how-to every time.
• Perplexity favors recent content with visible dates.

The pattern is almost boring: the fundamentals still carry. Rank for the query, structure the page so machines can resolve what it's about (schema markup is part of that), and publish things worth citing.

What I'm changing, and what I'm not

Changing: visible updated dates on every page, site-wide. Perplexity rewards them, readers trust them, and I had them on articles but not everywhere. Also changing my publishing mix. This article is the first in a deliberate original-data lane, because the citation research says that's the content type models reach for, and I'd rather publish my own numbers than paraphrase someone else's.

Not changing: robots.txt stays fully open, training bots included. Being in the training data of a system that millions of people ask for recommendations is a marketing channel, and I'm choosing it on purpose. The llms.txt file stays too, with my expectations parked at zero. And I'm not buying an AI-visibility tracking SaaS. Most of them are sampling model outputs and charging dashboard prices for it, and I can sample model outputs myself.

What this audit can't tell you

Fair warning before you run this on your own site: the method verifies access and index presence. It does not verify outcomes.

There's no Search Console for AI answers. I can't pull a report of every query where Claude cited me last month. Asking the assistants directly is sampling, not measurement, and the answers shift between runs because the models aren't deterministic. The Brave result in Step 3 is a strong signal, but it's one query on one day.

I also can't fully test verified-bot handling from my desk, as covered in Step 2. And none of this tells you what weight any signal carries inside a model's retrieval, because no provider publishes that. Anyone who claims they know the ranking factors of ChatGPT search is guessing with confidence.

What the audit does establish is the necessary conditions. If a crawler can't fetch your pages or your pages aren't in the index the model reads, no content strategy fixes that. Check the pipes first.

If you'd rather have me run this, my $500 SEO health check now includes every AI-visibility check in this article alongside the standard technical audit.

FAQ