The crawler landscape changed and no one sent a memo

Besides Googlebot and Bingbot, there is now a whole family of AI crawlers hitting your site for different reasons. For openai, GPTBot and OAI-SearchBot crawl . ClaudeBot is crawling for Anthropic. PerplexityBot crawls for Perplexity. Googlebot-Extended is a separate signal from regular Googlebot, specifically for AI training. Each one announces itself with its own user agent string, and each one can be permitted or blocked independently in your robots.txt file.

Here’s the part that surprises people. If you haven’t explicitly written rules for these bots, you’re not blocking them by default — you’re allowing them. A robots.txt file without AI-specific directives is not neutral. it's an open door There are plenty of sites that have opinions about whether that's a good thing or not, but very few site owners have actually made the decision on purpose. They just haven't looked. Most of them.

JavaScript problem gets worse, not better

If your site relies on a JavaScript framework to generate content on the client side, you know that traditional crawlers have struggled with that in the past. Googlebot has improved significantly in executing JS, but it’s never been instantaneous, and never guaranteed for every crawler.

This is something AI crawlers are typically a step behind on. Many of them don’t execute JavaScript the way a browser does, so any content that only appears after your React or Vue app hydrates may not even exist to that crawler. Headings, body text, structured data - if they are injected client-side and the crawler doesn't run your JS then none of it is seen and none of it can be cited in an AI answer.

The practical fix looks the same as it always has: server-side rendering, static site generation, or a hybrid approach that ensures the content a crawler actually needs — headings, core text, key structured data — is present in the initial HTML response, before any JavaScript runs. The difference now is that the list of crawlers that you need this to work for is longer, and some of them are less forgiving than Googlebot ever was.

A new file convention that you should be aware of

One newer piece of this puzzle is llms.txt — a file, similar in spirit to robots.txt, meant to give AI agents a clean, structured entry point into a site's content, separate from how your HTML happens to be organized for human visitors. It’s not yet a universal standard the way robots.txt is, and adoption is still early, but it’s showing up often enough in 2026 discussions that it’s worth having on your radar, especially if a meaningful chunk of your traffic is starting to come from AI-driven referrals rather than classic search clicks.

How to check where you are?

This is not something you have to guess at by staring at logs. Some concrete steps:

1. Compare your robots.txt to the latest AI user agent list — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and whatever else has been added since. Decide what you’re letting happen.

2. Simulate crawler access with no JavaScript. Load your key pages with JS off, or check the raw HTML response before hydration. If your headings and core content are missing, an AI crawler that doesn't run JS won't see it.

3. Check that structured data is present in the raw response, not just the rendered DOM. The same JS-dependency problem applies to schema markup as to visible text.

4. See if you actually have a llms.txt and if the content it points to is up-to-date. Doing this by hand, page by page, on anything larger than a handful of URLs isn't realistic—which is exactly why a full site crawl is worth running rather than spot-checking a few pages and hoping the rest follow the same pattern.

How this fits into a standard SEO workflow

The good news is that most of this isn’t a separate discipline from technical SEO – it’s the same fundamentals (clean HTML, working structured data, sane crawl directives) applied to a wider set of crawlers than before. The tools built for traditional tech audits are evolving to check for this exact thing: crawler-specific robots.txt rules, JavaScript-dependent content, structured data validity and llms.txt generation, on top of the broken links and duplicate content checks that were already standard.

For instance, KWT Spider 2.0, a desktop SEO crawler, has integrated these checks into its standard audit, generating a llms.txt file directly from crawl data and validating structured data across a full site, on top of its existing technical SEO checks.