Jul 30, 2026
How ChatGPT retrieves a rental catalogue
WHAT WE FOUND
- Answer-time demand arrived on day two. The first fetch triggered by someone's question landed 31.4 hours after the site went up, with no announcement, no backlinks and no traffic. It went straight to a neighbourhood page.
- Demand ramps, then settles. Weekly fetches grew from 15 to 545 over three weeks, then held near 48 a day. The ramp is discovery finishing. The plateau is the rate people were asking all along.
- It wants the aggregate, not the listing. 47.6% of fetches were neighbourhood pages against 28.3% individual units. The page that answers is the one holding a count, a price range and a median, computed before it asked.
- 68% of sessions are a single fetch. One neighbourhood page, question answered, done. Only a quarter of browsing sessions ever open a unit.
- It writes URLs it has never seen, but only out of words it was given. ChatGPT learned the filter grammar from the 42 neighbourhoods where we implemented it, then applied it to neighbourhoods where nothing advertised it, producing 20 failed requests. It never once invented a filter word we had not published, even for what it was plainly hunting: 70 of the units it opened were four-bedroom, and it never asked for a four-bedroom filter.
- It reads HTML and nothing else. Zero query strings, zero Markdown twins, zero sitemap or llms.txt fetches, across all 2,001 requests.
Scope
One agent, 41 days, 2,001 IP-verified fetches, read from CloudFront access logs we owned end to end. This report deliberately ignores crawlers. Training and indexing bots accounted for the large majority of the 295,710 requests this site served, and we cover that traffic elsewhere. Here we look only at ChatGPT-User: the agent OpenAI fires when a person has asked something and the model decides it needs a page to answer.
It is the third experiment in a series. The theatres edition asked what happens when you unlock a catalogue AI cannot reach. The companion report covers twenty live customer knowledge bases. This one asks a narrower question: when an assistant comes to a catalogue on someone's behalf, what does it actually do?
What we watched
In June we published openrentalsyyc.com, a catalogue of Calgary rentals republished from RentFaster and organized by neighbourhood. It is a plain, static site: no JavaScript, no search box, no filters that run in a browser. Every page is a flat document at a stable address.
| What it holds | 5,554 listings across 238 Calgary neighbourhoods, rebuilt daily from a fresh snapshot. |
| Four page classes | A home index, one page per neighbourhood, one page per listing, and 200 filter pages covering 42 neighbourhoods. |
| Every page, twice | Human HTML, and a plain-Markdown twin at the same path (/calgary/beltline.md). |
| Discovery | Sharded sitemaps, llms.txt, robots.txt. No promotion, no backlinks, no ad spend. |
Table 1. The catalogue as it stood at the end of the window. Counts are read from the storage bucket, not from the site's own summary line.
Over 42 days the site served 295,710 requests. Of those, 2,503 claimed to be ChatGPT-User, and 2,001 resolved to OpenAI's published address ranges. The remaining 502 were something else wearing the name, and are excluded from every number in this report.
That is the population: 2,001 moments when a person asked ChatGPT something and it decided this catalogue was worth opening.
First contact
The site went live on 18 June with no announcement. Crawlers found it within the hour: ClaudeBot at 40 minutes, OpenAI's GPTBot and OAI-SearchBot at 80 minutes, Googlebot at half a day.
Answer-time retrieval took a day longer. The first verified ChatGPT-User fetch landed at 31.4 hours, and it did not go to the home page. It went to /calgary/savanna/, a single neighbourhood, which means the model already had a specific question and had already worked out which page might hold the answer.
Figure 1. Log scale. Only agents whose vendor publishes IP ranges appear, because only those can be verified. Anthropic's answer-time agent, Claude-User, took 27.8 days and sent 12 requests in total.
The ordering matters more than the durations. Retrieval arrived before the training crawlers had finished their first sweep, which means a page can be answering a real person's question long before any model has been trained on it. Retrieval follows the search index, and the search index followed the sitemap within the hour.
The curve
Demand grew quickly for three weeks, then stopped growing.
Figure 2. 19 Jun to 29 Jul 2026. The final day is partial. The single spike is one session, not a demand event.
Weekly totals ran 15, then 90, then 357, then 422, then 545. After the first week of July the curve flattens at roughly 48 fetches a day and holds there, drifting neither up nor down beyond ordinary day-to-day noise.
A flat line is easy to read as a disappointment. It is more useful to read the two phases as measuring different things. The ramp is discovery: the weeks in which the model works out that the catalogue exists and is worth opening. The plateau is what was underneath all along, the rate at which people happen to ask an assistant about Calgary rentals.
Publishing changed how well those questions could be answered. It did not change how often they were asked. That second number belongs to the subject rather than the site, and it varies enormously between subjects: a catalogue covering something people ask about in concentrated bursts would produce a completely different curve without anyone publishing differently. Worth knowing before setting an expectation about what a new catalogue should do.
The one spike, 245 fetches on 19 July, is not a crowd. 191 of them are a single 22-minute session, and across that day and the next the page mix flips from browsing toward individual units. It is one person running one long search, which is the subject of chapter five.
What it asks for
Nearly half of every fetch was a neighbourhood page.
Figure 3. Bedroom counts are parsed from listing slugs, so units whose title omits a layout are not counted.
To understand why the neighbourhood page dominates, it helps to know what is on one. Here is how /calgary/cranston/ opens:
37 rentals are listed in Cranston, Calgary from $800/mo to $3,250/mo as of Wed Jul 29, 2026. The median asking rent is $2,000/mo. Available layouts include bachelor, 1-bedroom, 2-bedroom, 3-bedroom and 4-bedroom.
Then every unit, each with price, bedrooms, bathrooms, square footage and property type on one line. A count, a range, a median and a layout inventory, all computed before anyone asked.
That is the whole reason this page class wins. A question like "what do two-bedrooms go for in Cranston" is answerable from that first paragraph in a single fetch. The alternative, opening thirty-seven listing pages and doing arithmetic across them, costs thirty-seven round trips and invites the model to get the arithmetic wrong. The valuable page is the one that has already done the work.
When it does open individual units, the pattern is consistent. Of 566 listing fetches, 43% describe a house, bungalow or duplex, against 5% condos and 3% apartments. Three-bedroom and larger units outnumber one-bedrooms and studios roughly four to one. Garages, finished basements and walkouts recur. This is household relocation, not a single renter looking for a room, and it is worth noting because it is not what the neighbourhood mix alone would suggest: several of the busiest neighbourhoods sit near the University of Calgary.
Across the window it reached 174 of the 238 neighbourhoods we published, so roughly three-quarters of the city, and 447 distinct units. It is not sampling. It is working through the map.
Sessions
Server logs do not record who asked or why. What they do record is when, and the timing carries more structure than we expected.
Figure 4. The distribution is bimodal: fetches either follow each other within seconds, or after a long pause. Cutting between the two is what makes a session boundary defensible rather than arbitrary.
Cutting at five minutes yields 991 sessions. One caution before reading anything into them: OpenAI spreads a single burst of fetches across many addresses, up to 16 in one case, so grouping this traffic by IP would shred one session into a dozen fragments. Time is the only usable boundary, and a session here is a hypothesis about one person's question, not a fact the logs assert.
The headline is how short they are. 678 of the 991 sessions are a single fetch. One neighbourhood page, and nothing follows. Of sessions that continue, the median lasts 51 seconds, and only 53 of 215 browsing sessions ever open a unit.
Sixty percent of sessions open on a neighbourhood page, 24% on the home page, 13% on a unit. The model usually knows where it is going before it arrives.
The long sessions look completely different. The largest ran 191 requests over 22.8 minutes across 16 addresses:
Figure 5. Fetches sharing an elapsed time arrived together in one parallel burst. Colour marks the page class; red marks a request that returned 404.
The rhythm is breadth-first. It pulls a row of neighbouring communities at once, fans out across their units in a single burst, then moves to the next cluster and repeats. Four neighbourhood pages in one second, then five listings in the next.
The clusters are geographic. That session works through Calgary's deep south and southeast: Pine Creek, Midnapore, Bridlewood, Silverado, Cranston, Canyon Meadows, Woodbine, Lake Bonavista, Deer Run, Queensland. A different session sweeps the northwest instead: Scenic Acres, Hamptons, Arbour Lake, Edgemont. The model is decomposing something like "family homes in south Calgary" into its constituent neighbourhoods and traversing them in order, holding a map of the city it did not get from us.
It writes its own URLs
The failed requests turned out to be the most informative part of the log, because some of the addresses in them are addresses we never published.
81 requests failed, 4.0% of the total, in two groups with nothing in common.
61 were units that no longer exist. Sixty of those sixty-one are gone from the site entirely: leased, withdrawn, rotated out between the model learning of them and someone asking. Any catalogue whose records retire will collect these, because a model's picture of what exists always trails what exists. The rate follows how fast the inventory turns over, and here it stays low, 3.0% of all requests.
20 were filter pages ChatGPT constructed itself. Those are the interesting ones.
We publish filter pages at /calgary/{neighbourhood}/{filter}/, where the filter is one of five words: 0-bedroom, 1-bedroom, 2-bedroom, 3-bedroom or pet-friendly. We built 200 of them, covering 42 of the 238 neighbourhoods. On those 42, the filter links appear in the page itself. On the other 196, nothing on the page mentions filters at all.
ChatGPT requested filter URLs for neighbourhoods in the second group. It asked for /calgary/cranston/pet-friendly/ on three separate days, a week apart each time, and got a 404 every time.
Figure 6. Published counts are neighbourhoods carrying that filter page. Requests are split into those we served and those that 404ed.
Two things make us confident it learned this from us rather than guessing a common web convention.
First, the vocabulary matches exactly. Across 2,001 requests it used five filter words and no others, and those five are precisely the five we publish. There is no furnished, no parking, no 2-bed, no two-bedroom, no pets-allowed. A model working from general priors would scatter across synonyms. This did not scatter at all.
Second, 0-bedroom is not a phrase anyone would guess. The natural words are "studio" or "bachelor", and the page body itself says "Bachelor". 0-bedroom is an idiosyncratic token we invented, and it asked for it seven times.
It is not following links. It inferred how the site addresses things, then wrote addresses we never gave it.
That changes what a coverage gap costs. 134 of the 176 neighbourhoods it visited have no filter pages, which is 54% of all neighbourhood traffic landing somewhere it cannot narrow down. The model already believes those URLs exist. It keeps asking.
The transferable part is not the number, it is the mechanism. Any site that implements a pattern across part of its URL space is teaching that pattern as though it held everywhere. A model that has learned it will write addresses to match, and where the implementation stops, those requests fail. Partial coverage does not read as partial to an assistant, and the resulting 404s are the only visible symptom.
What it never tried
The 404s show what ChatGPT expected to find. The requests it never made show where the edge of its expectations sat.
Across 2,001 requests, ChatGPT used exactly four URL shapes: the home page, a neighbourhood, a listing, and a neighbourhood-plus-filter. It never attempted any of the following:
| A Markdown twin | Every page has one. It fetched zero, despite GPTBot, the same company's training crawler, taking Markdown on half its fetches. |
| A query string | Zero of 2,001. Never ?bedrooms=3, never a search parameter of any kind. |
Sitemaps or llms.txt | Never fetched, in either direction. It did not read our machine-readable descriptions of the site. |
| A filter word we don't publish | No 4-bedroom, no price band, no property type, even though 70 of the units it opened were four-bedroom and 242 were houses. |
| Two filters at once | No /beltline/2-bedroom/pet-friendly/. It never composed. |
| A city-level filter | No /calgary/2-bedroom/. The filter only ever appeared below a neighbourhood, exactly as we publish it. |
Table 2. Things that either exist on the site or would be reasonable to guess, and were never once requested.
Read together with chapter six, this is a sharp boundary. ChatGPT freely generalized along the axis where it had seen variation, swapping in neighbourhood names it had never seen paired with a filter. It did not generalize along the axis where it had only ever seen five fixed values. It invented /calgary/cranston/pet-friendly/ without hesitation and never once invented /calgary/beltline/4-bedroom/.
The four-bedroom case is the cleanest demonstration, because the intent was plainly there. Seventy of the units it opened were four-bedroom, and it was visibly hunting family houses. It still never asked for a four-bedroom filter, because we never showed it one.
The absence of Markdown fetches is worth separating out. Across this site, Markdown adoption splits cleanly by function: training crawlers take the twin around half the time, while search indexers and answer-time agents never do. Publishing Markdown twins is a training-corpus optimization. It does not appear to help the agent that shows up when a person is waiting.
What to take from this
One catalogue in one city is not a general law, and the honest caveats are in the methodology. But the behaviours below are mechanical rather than particular to rentals, and they are worth testing against any site with a machine audience.
Precompute the answer, not just the record. Nearly half of answer-time demand went to pages holding a count, a range and a median. A model asked what something costs in a given area can read that from one paragraph, or it can open thirty-seven records and do arithmetic it may get wrong. The arithmetic over your inventory, computed in advance and given a stable address, is a more useful artifact than the inventory itself.
Assume there is no second request. Two-thirds of sessions ended after one page. Whatever that page fails to say is unlikely to be asked again a different way, so a thin landing page is not the start of a conversation. It is the whole conversation.
A URL pattern is a promise. Implement it across part of your site and a model will infer it holds across all of it, then construct addresses accordingly. Consistency across a namespace matters more than depth in one corner of it.
Models recombine your vocabulary; they do not extend it. This one is the least intuitive and probably the most useful. An assistant will freely swap in values it has seen vary, and it will not invent new values for a dimension where it has only ever seen a fixed set.
That last point has an uncomfortable corollary. A 404 is a gift: a model telling you exactly what it expected you to have. But it can only ask in words you have already published. Whatever you left out, it never asks for, and you never find out.
Reading your own logs
None of this needed special tooling. If you have raw access logs, whether from CloudFront, Cloudflare, nginx or anything else that records a user-agent and a client IP, you can run the same analysis. A rough order of operations, and the traps we hit:
Filter to answer-time agents first. The user-agent strings that matter are ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher. They are a small fraction of AI traffic and the only tier where a person is waiting. Everything else, GPTBot, ClaudeBot, Googlebot, is crawling on its own schedule and answers a different question.
Verify against published IP ranges before you count anything. This is not optional. 20% of the requests claiming to be ChatGPT-User here were something else wearing the name, and on the same site a quarter of the PerplexityBot claims failed too. Every major vendor publishes prefix lists; resolve each request against them and discard what does not match. Two further traps: a vendor's lists can move or go stale, so re-fetch them at analysis time, and Anthropic's answer-time agent needs two lists combined rather than one.
Group by page class, not by URL. Individual URLs are too sparse to show a pattern. Collapsing them into three or four shapes, index, category, detail, filter, is what makes the aggregate-versus-record split visible.
Cut sessions on time, then check the cut. Plot the gaps between consecutive fetches before choosing a threshold. If the distribution is bimodal, cut in the trough and say so. Do not group by IP: OpenAI spreads a single burst across many addresses, and a request whose neighbours are seconds away is far more likely to belong with them than with anything sharing its IP.
Read the 404s last, and read them closely. Separate records that expired from addresses the model invented. The invented ones are a free list of what an assistant believes your site should contain, written in your own URL vocabulary, and they are the most directly actionable output of the whole exercise.
The one thing logs cannot tell you is what the assistant said afterwards. Everything here is what was fetched, never what was answered, and the gap between those two is where most of the interesting questions still sit.
Methodology
Source. CloudFront access logs for openrentalsyyc.com, 18 June to 29 July 2026: 295,710 request rows across 12,434 log files, parsed in full with no sampling.
Agent identification and verification. Requests are matched on user-agent substring, then every request claiming ChatGPT-User is resolved against OpenAI's published prefix list for that agent.1 2,503 requests made the claim; 2,001 resolved inside the published ranges and 502 did not. Only the verified 2,001 appear anywhere in this report. Figure 1 additionally verifies Anthropic, Google and Perplexity agents against their own published lists.234 Anthropic's answer-time agent requires combining two lists, its crawler publication and its platform outbound ranges; the latter endpoint was unreachable at analysis time, so its range is carried forward from our previous run.
Sessions. Sessions are cut on a 300-second gap between consecutive verified fetches, chosen because the gap distribution in Figure 4 is bimodal with a trough there. Session counts are stable across thresholds from 60 to 900 seconds. Sessions are not cut on IP, because OpenAI distributes a single burst across many addresses. A session is an inference about one person's question and should be read as such.
Definitions. A page class is derived from URL shape: home, neighbourhood (/calgary/{x}/), filter (/calgary/{x}/{y}/) and listing (/listings/{slug}/). Bedroom counts in Figure 3 are parsed from listing slugs, so units whose title omits a layout are absent. Catalogue counts in Table 1 are read from the storage bucket rather than the site's own summary line, which is regenerated daily and drifts slightly.
Limits. One property, one city, one vertical, 41 days, on a domain new enough that agent behaviour toward it may not resemble behaviour toward an established one. 2,001 requests is a small population, and the two largest sessions account for 15% of it. We measured what ChatGPT fetched, not what it said afterwards: no citation or answer-quality testing was run. Figures are requests, not users.
References
- OpenAI, overview of OpenAI crawlersPublished IP ranges and documented behaviour for GPTBot, OAI-SearchBot and ChatGPT-User.
- Anthropic, Claude crawler IP publicationThe prefix list used to verify ClaudeBot and Claude-User.
- Google, verifying Googlebot and other Google crawlersPrefix lists for Googlebot, special-case crawlers and user-triggered fetchers.
- Perplexity, PerplexityBot and Perplexity-UserPublished ranges and the distinction between indexing and user-triggered fetching.
- Courtyard, an experiment to unlock better agentic commerce for theatresSeven weeks of IP-verified logs on a national cinema seat-availability catalogue.
- Courtyard, how AI reads a knowledge baseThirty days of IP-verified logs across twenty live small-business knowledge bases.