How AI reads a knowledge base

The vantage point

Most of what's written about how AI assistants read the web is inferred from the output side: reading answers, counting brand mentions, guessing at what the model retrieved. That's a real approach, but it's mostly black box. Peec AI put it well recently: "Server logs are now one of the most direct ways to see why your AI search visibility is underperforming."⁶

Courtyard operates the CDN edge for a set of small-business AI knowledge bases, structured machine-readable sites hosted at ai.{customer-domain} that AI assistants can fetch to answer questions about the business. Because we're the origin, we see every request an assistant makes: the URL, the timestamp, the source IP, the user-agent, the status code, the response bytes. We do not see the user's prompt, the model's answer, or whether the fetched content actually shaped the response: the server log is a record of fetches, not of conversations. (We track citations and answers separately, through prompt monitoring across assistants; that's a different dataset, and out of scope here.)

What follows is what those server logs look like when you IP-verify them, cluster them into potential conversations, and hold them up against what OpenAI, Anthropic, Google, and independent researchers have said about how these systems are supposed to work.

The question users ask an AI about a fitness studio bears no resemblance to the question they ask about a dental clinic. One is "is this class booked?". The other is "do you take my insurance?". We deliberately kept vertical mix wide, twelve verticals across twenty subdomains, to avoid describing one industry's behavior as if it were everyone's.

The three functions of AI bots

Not every AI-adjacent user-agent means the same thing. Peec AI's writeup on server logs frames this well: AI bots do fundamentally different jobs, and if you conflate them, you draw the wrong conclusions.⁶ Grouping by function, not by vendor, is what lets you read the logs.

Three functions cover every AI-related user-agent we see, and they map cleanly onto each vendor's published documentation:

The functional split matters because the three groups have wildly different implications. A spike in training traffic might tell you a new model version is probably being prepared. A spike in indexing traffic tells you an assistant has decided your site is worth cataloguing. A spike in live-retrieval traffic tells you real users are asking about your business right now, and the assistant is going to the source to answer.

The next section shows the aggregate volume across all three functions on our sample. Then we look at each mode in turn: training (raw volume, coverage, cadence), indexing (how fast a new site gets picked up and re-crawled), and live retrieval (the real-time demand signal we spend most of this report on).

Total demand: how much is AI actually visiting?

Our origin serves millions of bot requests a week across customer subdomains. The overwhelming majority are training crawlers, indexing bots, and (as the spoofing section documents) a persistent stream of credential-scanning traffic wearing an AI-bot user-agent as camouflage. Live retrieval, the fetches we care most about here, is a small slice of that on every site we looked at.

The shape isn't unusual at our scale. Cloudflare Radar's AI Insights, drawing on their network-scale view across millions of domains, shows the same distribution: training and indexing crawlers dominate raw AI bot volume, while user-triggered live-retrieval traffic sits in the low single digits as a share of total AI bot activity.¹³ Once you subtract training and indexing, what remains is the interesting slice: real conversations happening now, resulting in a bot fetching a specific customer's page to help answer someone's question.

ChatGPT-User is the dominant live-retrieval user-agent on our sample. 1,143 raw fetches over thirty days, of which 1,044 IP-verify against OpenAI's published ranges (91.3%), from 468 unique verified IPs across 18 of 20 subdomains. The pattern matches everything OpenAI has publicly documented: one fetch per Azure worker, fresh IP each time, root-heavy target selection.

Claude-User live-retrieval, once you exclude Courtyard-internal MCP testing, is functionally zero. The un-excluded volume looks big, but nearly all of it is /mcp tool-call plumbing from our own team's testing, not user-driven retrieval (the mechanism and attribution are detailed later). Once that testing is stripped, external Claude-User traffic on our sample is 48 hits over thirty days across 17 subdomains, of which 85% are automated probes to discovery files (robots.txt, sitemap.xml, llms.txt) and none is a recognizable user-driven retrieval pattern.

Perplexity-User is present but none of it verifies, and on inspection it isn't real Perplexity at all. All 22 Perplexity-User hits in the window fail Perplexity's published perplexitybot.json, and every one is a credential-scanner probe from two IPs across 3 subdomains, not a content fetch. We draw no retrieval conclusions from it; it's user-agent spoofing, discussed alongside the other named spoofers in the spoofing section.

The asymmetry is the story. Both OpenAI and Anthropic ingest our fleet at training time. Only OpenAI (via ChatGPT-User) retrieves from it at query time on any material scale. The training and indexing sections cover those footprints in detail; from there the report narrows to live-retrieval, the fetches that fire because a real user asked an assistant a question about a specific business. On our sample that's overwhelmingly a ChatGPT story.

After IP verification, roughly one in twelve ChatGPT-User hits disappears. 1,143 total, 1,044 verified, 99 dropped

Training: what pre-training crawlers do

Training crawlers exist to gather content that will feed the next model version. They don't answer users directly. Nine named training crawlers appear consistently in our logs. Their behavior is different in kind from live retrieval: systematic, high-volume, cadence-driven rather than user-driven.

ClaudeBot is the dominant training crawler on our fleet, by an order of magnitude. 15,201 fetches over thirty days from 288 unique IPs, across all twenty subdomains in the sample. About 65% of that volume is robots.txt + sitemap.xml discovery, the rest is deep-page enumeration. That corroborates the earlier finding: Anthropic is systematically ingesting our fleet for training, even though its live-retrieval footprint is functionally zero.

Everything else is much smaller in volume but still meaningful. Bytespider fetches from 729 unique IPs, an unusually distributed pattern that suggests aggressive parallelization. GPTBot is comparable to Bytespider in volume but far more concentrated (102 IPs), consistent with a smaller, more disciplined pool. Meta uses two distinct crawlers on our fleet: the general-purpose facebookexternalhit (543 hits, link-preview and AI-facing content ingest) and the AI-dedicated Meta-ExternalAgent (203 hits). Amazon's Amazonbot is also present at 430 hits.

HTML vs. markdown consumption

The .md-twin story for training is more measurable than for live retrieval. When a customer publishes structured markdown variants alongside HTML pages, several training crawlers pick up the markdown preferentially. Amazonbot is the standout on the sample, with 21% of its total fetches (91 of 430) hitting .md variants directly, the highest ratio of any crawler we track. Meta-ExternalAgent hits .md at 18%. ClaudeBot and GPTBot pull them at lower rates, but consistently. Live-retrieval user-agents by comparison hit .md at ~3% (more below). If your goal is to influence what ends up in the next model version, publishing a clean .md variant of high-signal pages is measurably productive.

The more useful question here isn't the obvious discovery surfaces (sitemap.xml and robots.txt get pulled at scale by every training crawler, as you'd expect) but whether the newer AI-native formats get ingested for training at all. On our sample, they don't. Across every training-crawler fetch in the window, llms.txt, llms-full.txt, and ai-catalog.json drew zero, not one training crawler touched an orientation file. The only structured format they actually consume is the .md twin, pulled by GPTBot, Amazonbot, ClaudeBot, and Meta-ExternalAgent as shown above. If your goal is to shape what the next model version learns, the leverage is a clean .md variant of your high-signal pages; the llms.txt family, on this sample, is not what pre-training crawlers are reading.

Indexing: what search-index crawlers do

Indexing crawlers build the search corpus each AI assistant consults at query time. They differ from training in cadence (incremental and fast rather than batch) and purpose (freshness rather than comprehensiveness). This is the mode that determines whether your site is available to be picked as a candidate URL when a user asks the assistant a question.

Disclosure: what we push to indexers

Every number in this section reflects a mix of vendor-initiated discovery and Courtyard-solicited traffic. As part of onboarding, we actively push new customer URLs to the indexes that publish a submission surface: IndexNow (which fans out to Bing, and by extension into OpenAI's search step), Google's URL Inspection / Search Console indexing endpoints, and Brave Search's URL submission API. We do not (and cannot) push to training crawlers, and there is no equivalent submission surface for live retrieval, so those two functions are unaffected. Read the indexing numbers as "the indexer is aware of these URLs" (partly because we told it), not as pure organic crawl rate. The relative shape between indexers is still meaningful — every vendor got the same push, so the differences in what they do with the URLs reflect the vendor's behavior, not our submission strategy.

Googlebot and Bingbot are neck-and-neck on our sample. Googlebot at 4,692 fetches from 413 unique IPs across all 20 subdomains. Bingbot at 4,556 fetches from 415 unique IPs, also all 20 subdomains. The two crawlers arrive at roughly the same cadence and reach the same coverage. This is the empirical basis for the "OpenAI's search step uses Bing" story: OpenAI indexes our fleet via Bing at Google-scale, and ChatGPT queries that index at scale when it answers a user (see the anatomy of a fetch, later).

OAI-SearchBot appears within about two days of launch on our sample, consumes fewer pages, and re-crawls less aggressively than either Google or Bing. Applebot is present but small: it appears on most subdomains and typically consumes only sitemap.xml and homepage. PerplexityBot behaves like a smaller Bingbot: reasonable initial pickup, moderate re-crawl cadence, respectful of robots.txt. Bravebot (Brave Search / Leo AI) shows up at 144 hits across all 20 subdomains, hitting root and discovery files predominantly.

Markdown twins: small share, telling ratio

Indexing crawlers pick up .md variants too, but at low volume: markdown twins are only about 2% of indexing traffic overall. The signal worth noting is relative appetite, not absolute volume. OAI-SearchBot fetches the .md variant at roughly 23% of the rate it fetches the equivalent HTML deep pages, the highest ratio of any indexer on our sample (Bingbot ~10%, Googlebot ~2%). That's a real if modest hint that some search-index crawlers prefer structured content when it exists, which matters because that index is what AI assistants query later.

Coverage over time

The average subdomain reaches 90% Googlebot coverage in about 48 hours, 90% Bingbot in about 72 hours, 90% OAI-SearchBot in about five days, and 90% PerplexityBot in about a week. Applebot rarely exceeds sitemap-and-homepage; its coverage grows slowly because Apple has not yet meaningfully signalled that the underlying search index feeds a live AI product.

The practical implication: if you are watching for evidence that a new site is being picked up by AI search, look for OAI-SearchBot within two days and PerplexityBot within about a week. If you don't see them by day ten, something in the site's setup (robots.txt, sitemap health, DNS/CDN) is likely blocking discovery.

The spoof problem: trust, but IP-verify

A meaningful share of what claims to be an AI bot is credential-scanning traffic using the user-agent as camouflage, hoping the target site has an allowlist for "AI bots" that would let a probe past a WAF rule.

The pattern isn't specific to our fleet. Cloudflare's bot-management research has flagged AI-UA spoofing as a growing threat surface, driven directly by the wave of WAF rules that whitelist AI crawler user-agents to preserve indexing signal.¹³ The ai.{customer} naming pattern almost certainly amplifies the effect on our origin (see the Scope note above), so the numbers below should be read as an upper bound on SMB apex spoofing rather than a population estimate.

Two lists, or you miss the tool-call path

Anthropic reversed a two-year policy in May 2026 and began publishing an IP range list at claude.com/crawling/bots.json, moving from its old position, "we don't publish IP ranges because we use service provider public IPs", to a canonical list of Anthropic source IPs.¹¹

Here's the wrinkle. That bots.json list covers 20 IP prefixes, mostly Google Cloud (34.x, 35.x) and a small 216.73.216.0/22 block. It correctly verifies the crawler code paths at Anthropic: ClaudeBot, Claude-User with a canonical Mozilla-style user-agent, Claude-SearchBot. But it does not cover Anthropic's Platform outbound range 160.79.104.0/21, which is where API tool calls (including MCP, web_fetch, and Claude Code CLI) exit. On our sample, ~96% of raw traffic bearing a Claude-related user-agent comes from the Platform range, not the crawler range (nearly all of it our own MCP testing). Analytics tools that only check bots.json will show that traffic as unverified.

Named spoofers

Below the vendor-list mismatches, we have real bad actors. Three source IPs repeatedly hit customer subdomains under an AI-bot user-agent, consistently fail every vendor's IP check, probe credential-adjacent paths (/.env, /gcp-credentials.json, /.git/config), and are almost always returned a 403. They are the most persistent offenders in the ~9% of ChatGPT-User hits that fail verification; the rest of that miss is a long tail of one-off IPs, including IPv6 sources that OpenAI's published list does not cover.

The probe surface is wider than /.env

The three serial spoofers are the visible tip. When we widened the classifier beyond the obvious /.env, /.git/, and /wp- signatures to catch the modern secret-file family (secrets.json, appsettings.json, firebase config, service-account keys, /api/env, actuator/env, .npmrc, .pypirc, .vscode/settings.json, .cursor/mcp.json), the probing volume roughly doubled in every mode. Over the thirty-day window on our top-20 sample, credential-probe requests bearing an AI user-agent went from 32 to 65 under the retrieval user-agents, 98 to 153 under training crawlers, and 110 to 183 under indexing crawlers. Same logs, same window. A naive filter that only greps for .env undercounts the probing by about half. Note this counts traffic by its claimed user-agent: none of these probe requests IP-verify, which is exactly why they are absent from the verified retrieval flow in Figure 6.

The point isn't that these attempts succeed. Almost all draw a 403, and an AI knowledge base is a static content surface with no secrets to leak. The point is what it says about the traffic mix: any origin that answers an AI user-agent is also probed by opportunistic scanners wearing the same user-agents. In the raw retrieval-UA stream, before IP verification, these probe requests outnumber every named buyer-intent path except /availability and /booking, then wash out entirely once you check source IPs. If you are reading your own access logs to understand who is fetching your knowledge base, the credential-scanning slice is larger than a first pass will show, it travels under the same ChatGPT-User and CCBot labels as the legitimate fetches, and only IP verification separates the two.

Anatomy of a live-retrieval fetch

What actually happens when a user asks ChatGPT "what times does this yoga studio open on Sundays?" and the studio has an AI knowledge base? The answer requires understanding a two-stage pipeline that OpenAI has documented obliquely and independent researchers have reverse-engineered.⁴

Stage one is the model's internal search tool, web.run in OpenAI's system, which fires fan-out sub-queries against OpenAI's own index (built on Bing plus their own crawl). This step happens entirely inside the model's compute. It doesn't touch our origin.

Stage two is where our logs come in. From the candidate URLs the search step returned, the model picks one or more to actually visit. That visit is what shows up in our logs as ChatGPT-User. The distinction matters: most of the model's "search" never touches the origin, so a server log only ever sees the small subset the model chose to open, not everything it considered.⁵

Search Engine Land measured how many fan-out queries the model runs before fetching. GPT-5.3 Instant fires two or three. GPT-5.4 Thinking fires roughly ten to forty depending on the depth of the question.⁴ Every one of those sub-queries can produce zero, one, or many URL fetches downstream. That's why a single user question can produce a burst of fifteen rapid hits to a small customer's site: the model is testing multiple retrievals in parallel to confirm one thing.

The pattern the visualization shows is characteristic. A reasoning-model conversation starts with a fan-out burst: several parallel fetches within a few seconds of each other, hitting the same host from different IPs in the Azure worker pool. Then a brief pause, followed by one or two confirming fetches, often to the same page the user's answer will end up citing.

The IP allocation is telling. Across all ChatGPT-User traffic in our sample, the ratio of distinct source IPs to fetches sits around 1:1.2. Almost every fetch comes from a fresh Azure worker. This is not a session-pinned model where one IP represents one user, every worker is ephemeral. Which means "unique source IPs" is a poor proxy for "unique users". Estimating real conversations from server logs alone requires time-clustering, not IP-counting, and even clustering only approximates: on higher-traffic sites, separate conversations about the same business within a few minutes collapse into one session.

Claude-User works differently. Because the traffic splits between crawlers (canonical UA, Google Cloud IPs) and the API's web_fetch tool (bare UA, Anthropic Platform IPs), the same conversation can produce fetches from two different code paths at Anthropic. Session clustering has to be done per user-agent flavor to stay honest.

Where AI goes: path-level analysis

If the last section was about when a fetch happens, this is about where. The path AI picks reveals what the user asked, and it reveals whether the model found a specific answer or fell back to the homepage.

Homepage first, then the buyer-intent pages

Just under two-thirds of all verified AI fetches (63%) land on the site's homepage. That's a retrieval-layer artifact: entity queries like "tell me about this business" resolve to the domain apex more often than to any specific deep page, and Bing's URL selection biases toward homepages for brand-anchored queries. The AI fetches what the search step recommended.

Two independent measurements corroborate the direction. Suganthan Mohanadasan's network-traffic teardown of ChatGPT sessions found the model's own result_source label frequently marks the homepage as the primary retrieval target for local-business queries.⁵ And Profound's output-side citation research on brand-anchored queries reports a similar apex-lean pattern before AI drills into deep pages for follow-up questions.¹⁴ The two vantage points, server-side (ours) and output-side (theirs), triangulate on the same behavior.

The shape of the remaining 37% is where the interesting signal lives. It doesn't spread evenly across a site. It clusters tightly on a small set of buyer-intent paths, the same pages a real customer would open if they were deciding whether to book:

• /availability (106 fetches, the single biggest deep path). The "can I get in?" page, appointment and class-schedule availability. Concentrated on the booking-driven verticals (physical therapy, fitness, dental, optometry). When AI fetches this, a ready-to-act customer is asking whether they can actually get a slot. It's a stronger buyer signal than any other page.
• /booking (74 fetches). The step after availability. Present on almost every non-restaurant vertical; often fetched in the same session as /availability.
• /pricing (49 fetches). The single most guarded page across the sample, and one AI still fetches when a user is comparison-shopping.
• /services (46 fetches). What the business does, in the machine's words. Fetched heavily on health and professional-services verticals. (Includes treatments, programs, and advisory-service pages.)
• /team or /practitioners (32 fetches). Longer tail, correlates with high-consideration verticals (physical therapy, dental).
• /menu (29 fetches). Restaurant-only. Fetched alongside /reservations in fan-out bursts.

The headline here is that /availability tops the list. AI isn't just fetching what a business is (services, team); it's fetching whether the customer can act right now. Actual opening-hours pages, by contrast, barely register (2 fetches) — the "are they open?" question is answered from the homepage or a structured field, not a dedicated page.

A long tail rounds out the picture. Agent-facing markdown twins (/menu.md, /programs.md) draw 30 verified fetches (2.9% of verified retrieval), evidence that some assistants pull the structured variant when it exists. Another 15 verified fetches land on pages outside the standard buyer-intent buckets, not low-intent, just specific to the business: /accessibility, /media, a couple of HVAC sizing tools, and, most of all, one law firm's practice-area pages (/corporate-services, /regulatory-compliance, /cross-border-structuring). That last cluster is the tell: those aren't generic pages, they're the specialized knowledge the owner chose to publish, and AI reached straight for it. Assistants pull the expertise that's specific to a business, not just its templated pages, so the "other" bucket stays per-business by nature, and that specificity is the signal, not noise. Two things it did not include are worth noting: no /about page drew a single verified fetch in the window, and these content-page hits sit almost entirely on one smaller subdomain, which is why the "other pages" tier in Figure 6 is fed almost entirely by a single professional-services host. Discovery files (robots.txt, llms.txt, llms-full.txt, sitemap.xml) all but vanish from verified retrieval (2 fetches): the discovery-file volume that shows up under retrieval user-agents is indexing and scanning traffic that fails IP verification, which is why it doesn't appear in Figure 6.

A scoping note on those numbers. Our sample is SMB directory-style content, hours, menus, services, pricing, which maps cleanly to specific URLs, so a .md twin buys little over the HTML it mirrors and users rarely ask anything llms.txt would answer. Technical documentation is a different world: users ask "how do I implement X", and Mintlify reports .md hit rates well into the double digits with llms.txt as a primary orientation surface.¹⁵ Read "3% on SMBs" as a statement about this content shape, not about agent-facing markdown in general.

The granularity is what makes this actionable. Reporting "AI fetched your /pricing page 50 times this month" is a business signal, "AI reads your knowledge pages" is not. The pages driving buyer decisions are the pages assistants are consuming to answer questions about them.

The MCP-discovery story: infrastructure works, organic demand hasn't started

Anthropic's crawlers periodically probe /.well-known/mcp.json and /.well-known/ai-catalog.json on the customer subdomains that expose them. These are indexing-crawler discovery probes, not conversational retrieval, and they're the closest thing to a live "AI is looking for your MCP endpoint" signal we see in the logs.¹²

We do see a large volume of hits on /mcp itself, from three tool-call surfaces: Anthropic Platform (Claude-User, ~2,000 hits), Anthropic Platform (claude-code, ~2,500 hits), and OpenAI Azure (openai-mcp, ~300 hits). Every one traces to Courtyard-internal testing, and all three are excluded from every other number in this report. The honest reading: organic external MCP discovery has not started on our fleet at any material scale we can attribute.

The absence is the finding, and it reads as a leading indicator. The infrastructure works. Claude Desktop treats third-party MCP servers as first-class. Our customer MCPs load and answer questions correctly. Codex-in-cloud can hit them. What hasn't happened yet is assistants proactively probing customer subdomains for MCP endpoints or discovery files outside a user's explicit connect action, in numbers that we can distinguish from our own testing.

Behavior differences across assistants

Put the three retrieval assistants side by side and the differences are stark, largely because only one of them has a real fingerprint to read at all.

Only one column describes a real assistant. ChatGPT-User matches what OpenAI has described publicly, an Azure worker spun up per fetch, a fresh IP almost every time, the homepage before deep pages, and Cloudflare's network-scale data shows the same dominance in every country where people ask assistants about local businesses.¹³ The other two columns are noise wearing a badge: Claude-User's 48 external hits are discovery-file probes from a single non-Anthropic scanner IP, and Perplexity-User's 22 are credential-scanner probes from two, neither a verified retrieval signal to characterize. Part of Claude's absence is likely subdomain-age (see the Scope note): new ai.{customer} subdomains underindex against long-lived apexes, which caps how far to generalize the "Claude barely retrieves us" claim beyond early-lifecycle sites.

One structural note worth naming here. Brave's Leo assistant is (in part) Claude-powered, but Leo does not issue Claude-User web_fetch calls against origins like ours at query time. It retrieves through Brave's own search index, populated by Bravebot (see the indexing section). Any Leo-via-Claude user activity on our fleet routes into the Bravebot line item, not into Claude-User. And as noted there, some of the Bravebot volume itself is solicited by Courtyard-submitted URLs, so it isn't a clean organic-demand signal either. Net effect: the Leo-via-Claude retrieval path is essentially invisible in a straight Claude-User count, and doesn't rescue the "Claude retrieves us live" story.

Google-Agent: still zero

Google announced Google-Agent as the user-triggered variant of their agentic browsing product. We check for it daily against the user-triggered-fetchers-google.json published IP list, and as of publication we've never seen a single request. Cloudflare's data shows Google adoption of AI-facing crawlers has grown steeply through 2026, so the pattern isn't going to stay at zero, it just hasn't hit our fleet yet.¹³

Going live: what a knowledge base sees on day zero

One of the more distinctive things about running a fleet of AI knowledge bases is that you see what happens when a brand-new one goes live. New subdomain, new content, no prior AI awareness. Then you watch the discovery mechanics unfold.

The sequence is consistent across our sample. The first bot through the door is usually a training crawler, not a search indexer: on the launches we can cleanly time, ClaudeBot shows up within an hour or two, right behind the very first robots.txt fetch and often ahead of the search engines. Anthropic's crawler is conspicuously eager. Indexing crawlers arrive in the same early window: Bingbot and Googlebot show up within a day, usually via robots.txt and sitemap.xml, with OAI-SearchBot and PerplexityBot following within a day or two. Live-retrieval traffic, meaning real user conversations, takes noticeably longer, and it doesn't correlate directly with indexing completeness. Some sites are being live-retrieved before they're fully indexed. Others are indexed for weeks before their first verified ChatGPT-User hit.

Discovery is fast. A robots.txt fetch on the same day as launch happens on more than three-quarters of the sample. Indexing is fast enough that Bingbot generally shows up within a day and Googlebot within two. Live retrieval, the point at which AI is actually answering someone about the business, is variable. Six days is the median, and the range spans one day to more than three weeks. What differs between fast and slow customers is whether real users are already asking assistants about that business, not the technical setup.

On customers that publish .md twins of their pages, the twins get picked up within about four days of launch.

Signal quality: what to trust, what not to

Not every log line is equally informative. Some tell you the exact user intent behind a real conversation; some tell you nothing beyond "a bot passed through"; some are actively misleading. Here's how we've come to sort them.

High signal. A ChatGPT-User fetch, IP-verified, to a specific deep page (/booking, /pricing, /availability): a real conversation with high-intent context. Retrieval of a .md twin variant: structured content being consumed. A multi-fetch session where three or more deep-page fetches happen within thirty seconds from the same IP block: a reasoning-model fan-out around a specific query.

Medium signal. IP-verified fetches to the domain root: a real conversation, but the underlying user question was probably brand-level or open-ended. Indexing traffic (OAI-SearchBot, Bingbot, PerplexityBot): meaningful for reach, not for real-time demand. Anthropic crawler probes to .well-known/*: genuine AI discovery attempts, not conversational.

Low signal, or noise. Any user-agent-only bot count without a corresponding IP-verification pass: the ~9% inflation is real. Cloudflare-CDN IPs claiming AI-bot user-agents: probably users on a privacy proxy, occasionally bot infrastructure. Aggregate "AI traffic" reports that don't split live retrieval from indexing from training. Cross-vendor comparisons where the underlying verification method differs.

What we still can't answer

The server-side vantage has hard epistemic limits. It's the only place you can see what AI actually fetches versus what it's documented to fetch, but it can't see the answer generation on the other side. Three open questions sit beyond it.

1. How does retrieval actually shape the answer? A fetch means the model considered our content, not that it used it. Seeing both halves of the exchange means pairing this server-side view with output-side citation and response analysis, which is exactly the work we're doing now, with research to share soon.
2. When, if ever, do the new AI-native standards get consumed, and by whom? ai-catalog.json, .well-known/mcp.json, and emerging agent-skills manifests are all bids to become a machine-readable front door, and right now almost nothing fetches them. The open question is which function picks them up first, indexing crawlers building a catalogue or live-retrieval agents at query time, and which vendors lead.
3. Can we identify the next wave of clients? A widening field of models, Grok, DeepSeek, Kimi, and others, will each arrive with their own fetchers and user-agents. Whether they announce themselves and publish verifiable IP ranges, or show up unattributable, is the question most likely to complicate this kind of log analysis.

If you work on any of these, or you sit on data that can answer them, we'd like to hear from you.