Original research

We checked 12 real local-business websites for AI-crawler access — here's what we found

No invented "400+ businesses" marquee. A small, disclosed sample, checked with the exact same tooling that runs on every real Citely audit — zero LLM calls, fully reproducible.

Published 2026-08 · Citely Research · Sample size: 12

We used to run a marquee on the homepage claiming we were "watching four answer engines for 400+ small businesses." That was placeholder copy, not a real number, and we've taken it down. Here's an actual small study we ran to replace it with something true.

What we did

We picked 12 real, public local-business homepages via plain web search — three each across plumbing, family dentistry, roofing, and personal-injury law, all serving the Austin, TX market. We excluded directory/aggregator listings (Yelp, Super Lawyers, etc.) and kept only businesses' own sites. This is not a scientifically representative sample — 12 sites in one metro area is a spot-check, not a survey — and we're not naming the individual businesses here, since the point is the pattern across the sample, not calling out any one company.

For each site we ran the two checks that don't require a live LLM call: a crawlability check (does robots.txt block any major AI crawler — GPTbot, OAI-SearchBot, ClaudeBot, Google-Extended — plus heading structure and schema.org markup) and a content-quality scan (does each section open with a direct claim or a filler sentence, does it include a concrete number). Both are the same modules the product runs on every real customer audit, not a separate demo script.

What we found

1 / 12

sites fully block GPTbot, ClaudeBot and Google-Extended in robots.txt — invisible to those assistants regardless of content quality

4 / 12

have zero structured data (schema.org) anywhere on the homepage

1 / 12

homepage has no <h1> at all

43%

average share of page sections that open with a concrete number in the first ~60 words

One law firm's site in our sample disallows ClaudeBot, GPTbot, and Google-Extended under User-agent: * in robots.txt — almost certainly an unintended side effect of a "block AI scrapers" setting, not a deliberate choice to be invisible to Claude, ChatGPT and Google's AI features. This is exactly the failure mode our crawlability gate checks first, before looking at content quality at all: no amount of good copy fixes a site an assistant was never allowed to fetch.

The more surprising result, honestly: content quality in this small sample was better than we expected going in. Every site we could fetch had a majority of sections opening with a direct claim rather than throat-clearing ("in this article, we'll cover…" style openers) — we don't want to overstate a problem that, at least in this sample, wasn't the biggest gap. The bigger, more consistent gap was structured data: a third of the sample has none at all, which means an assistant has no machine-readable signal for what kind of business this is, what it charges, or where it operates — it has to infer everything from prose.

Full per-site results

Vertical	Blocks an AI crawler	Has schema.org data	Has H1
Plumbing	No	No	Yes
Plumbing	No	No	Yes
Plumbing	No	Yes	Yes
Dentistry	No	No	Yes
Dentistry	No	Yes	Yes
Dentistry	No	No	No
Roofing	No	Yes	Yes
Roofing	No	Yes	Yes
Roofing	No	Yes	Yes
Legal	Yes — 3 bots	Yes	Yes
Legal	No	Yes	Yes
Legal	No	Yes	Yes

This 33% figure is what backs the "measured" stat on our homepage — it's ours, we ran it, it's reproducible. See the companion research post for the current 2026 third-party research on brand-visibility gaps. We're not yet claiming a measured before/after result of our own; we don't have enough paying customers to compute one honestly.

Methodology & reproducibility

Sample selected via plain web search for "[vertical] Austin Texas official website," first three non-directory results per vertical, four verticals, n=12 total.
Crawlability check: one GET to /robots.txt, parsed for a blanket Disallow under GPTbot / OAI-SearchBot / ClaudeBot / Google-Extended or User-agent: *. One GET to the homepage for heading structure (raw <h1>-<h6> tags) and JSON-LD structured-data extraction.
Content-quality check: homepage HTML split on <h2> boundaries; each section's opening ~60 words scored for filler-opener phrasing vs. a direct claim, and for presence of a concrete number.
Zero LLM calls anywhere in this study — every number above is a deterministic parse of public HTML/robots.txt, reproducible by anyone.
Run 2026-08. Individual business names withheld by design — the aggregate pattern is the point, not any one company's setup.