We published a new page and timed how long AI crawlers took to read it. The first fetch came six minutes after publish. What that means for your site.
Most of us still think about web crawling on Google time. You publish a page, you wait, and some days later the index catches up. This week I watched that mental model die in a server log: an AI crawler read a brand new page six minutes after I hit publish.
Six minutes. I sat there assuming the logs were wrong.
I wanted to know how fast AI assistants can actually read new content, so I set up a simple test. Publish a fresh page on an established domain. No sitemap ping, no manual submission, no links pointing at it from anywhere. Then ask ChatGPT, Claude and Perplexity a question that only the new page could answer, and watch which crawlers show up.
I expected hours for that first fetch. Maybe days. There is no waiting period anymore, and that changes a few things if you run a website.
Your customers already ask AI assistants about you. What your product costs, whether you integrate with the tool they use, how you compare to the alternative they found first. When that question gets asked, the assistant does not check a stale index from three weeks ago. It often fetches your page right then, live, while the customer waits for the answer.
That cuts both ways. Fix an outdated price or a wrong claim on your site and the very next AI answer can reflect it. Leave a gap, and the assistant fills it from whatever else it finds about you, which is usually a competitor’s comparison page.
Here is the part most site owners get wrong. “AI crawler traffic” is really three different kinds of traffic, and each identifies itself separately:
Live answer fetchers like ChatGPT-User read pages in real time to answer a user’s question. This is the six-minute path.
Search index crawlers like OAI-SearchBot build the indexes behind AI search features. These arrived within hours in my test.
Training crawlers like GPTBot collect content for training future models. This is the slow lane, and the only one most people mean when they say they “blocked AI”.
Watching the fetches come in taught me a few practical things about what gets read and what gets used.
Direct answers win. An on-demand fetcher grabs your page, skims it and quotes what it needs within seconds. A heading that matches the question, with the answer in the first paragraph under it, gets used. Pages that bury the answer get skipped. We see the same pattern with the AI agents we build at SiteSpeak: the content that resolves a visitor’s question in one step is the content that gets surfaced.
Your content has to be in the HTML. These fetchers do not reliably wait for JavaScript to render. If your site is a client-side app shell, the fastest and most valuable crawl path sees an empty page.
Updates count immediately. The assistant reads the current state of your page every time it fetches. Publishing and correcting are no longer things an index eventually notices. They land on the very next answer.
Six minutes is not much time to make a first impression. Make sure the page an assistant reads is one you would be happy to hear quoted back to a customer.
