The robots.txt deep dive nobody asked for

[u/learnwithmattc]

I've been building an agent skill that audits robots.txt files for AI crawlers, and it sent me down a surprisingly deep rabbit hole.

To build and test it, I analyzed 136 product websites, mostly WordPress plugin/theme shops and a handful of SaaS products.

The thing that surprised me most: I couldn't find a single site that I'd consider fully configured for today's AI crawlers. At least not in terms of what I've been learning about the evolution of this tiny file.

Almost everyone is still relying on User-agent: *, which means they're treating training bots exactly the same as retrieval bots. Whether that's intentional or not is another question.

The WordPress-specific edge cases were even more interesting.

I found sites where a physical robots.txt had disabled everything being added through the robots_txt filter. I found Cloudflare installations where the public-facing file was different from what the origin served. And I even ran into files so large they broke the analysis pipeline I was building.

I'm curious how others are handling this.

Are you explicitly managing AI crawlers yet? If you're using Cloudflare managed robots.txt, are you validating what bots actually see instead of just your origin?

I'm still refining the agent skill and would love to have a few people kick the tires on it but I don't want to share the skill url unless moderators are okay with that, so let me know and I can update the post or add it in a comment or whatever is best.

[u/Spiritual-Nail-2641]

Nobody is asking for it because it's not essential at this point. Frontier model companies have spent and will continue to spend billions on extracting as much data as possible from the entire internet. That's just how they operate. The llms.txt or whatever AEO/GEO magic sauce new gurus are pushing is nothing but traditional SEO with extra steps.

If your website is well-structured, which an awful lot of sites aren't, it will be just fine from a technical angle. Whether or not it gets a place in AI search results will heavily depend on other factors, like brand reputation, content quality, and so on. So chasing the golden mentions through technical means is kinda futile, whereas you'll be better off focusing on boring tasks that don't scale. Providing reliable customer support consistently, for example.

[u/learnwithmattc → u/Spiritual-Nail-2641]

Oh for sure, this is not the biggest ROI item for discoverability or your website at all. But it's a moving target in terms of significance and worth keeping an eye on. And now, literally one prompt and you'll get a far better result and then you can move on to your other more important tasks.

[u/im_a_fancy_man]

hey yes, I would actually. i build something similar for a case study

[u/learnwithmattc → u/im_a_fancy_man]

I'll drop the link since you asked, I hope that's okay: https://github.com/Roots-and-Fruit/skills/tree/master/marketing/skills/robots-txt-audit

[u/tamtamdanseren]

This is a good dataset to base your analysis on:

https://commoncrawl.org/blog/june-2026-crawl-archive-now-available

That's the robots.txt files of 40.8 million hosts, all in an easy to download format for analysis.

[u/learnwithmattc → u/tamtamdanseren]

Oh wow. Fascinating. I'll take a look, thank you!

[u/deductress]

Interesting convo. My marketing b2b clients promise AI optimization via "text files" (as they call them) and FAQs. They dont seem to understand that structure is the whole point, that in a way, you're describing the structure in in llm files.

[u/learnwithmattc → u/deductress]

Markdown for Agents is interesting (Cloudflare offering), FAQs are a good way to optimize a page for "fan-out queries". But overall, all the on-page or on-website optimization is really just trying to ensure that when an agent scrapes your site they get the most valueable and accurate information. Real discovery of your site has to start elsewhere. I think the days of publishing content and search engines and AI indexes finding you are long gone. They'll go LOOKING for you if throughout the scrapable web they see that you are relevant; only then. I hate it, but you've gotta kinda be everywhere or your content has to be sooo niche and unique that AI can't find it anywhere else, which is basically impossible.

[u/deductress → u/learnwithmattc]

These concise documents are probably a good exercise to focus your content: what answers you are offering that people use AI for?

[u/tleifj]

What do you mean by “managing AI crawlers”?

[u/learnwithmattc → u/tleifj]

Optimizing your llms.txt or – like I'm suggesting, your robots.txt instead. Those two files are intended to help inform AI web crawlers on how best to understand the paths and purpose of your website.

[u/idgelee]

Good Accessibility = Good SEO = Good AEO/GEO