Are ai.txt and llms.txt becoming the new robots.txt?
Over the past few months, I’ve been analyzing how AI answer engines discover and interpret websites.
We’re starting to see a pattern:
• robots.txt controls crawling
• sitemap.xml helps discovery
• structured data helps search engines
But AI systems introduce something new:
AI-ready artifacts like:
ai.txt
llms.txt
ai-feed.json
structured RAG indices
Some companies are already publishing these.
Most are not.
My questions to builders and founders here:
Are you exposing structured artifacts specifically for AI systems?
Have you tested whether ChatGPT / Copilot / Perplexity can actually discover them?
Do you think ai.txt / llms.txt will become a standard like robots.txt?
Feels like we’re early in defining the “AI web layer.”
Curious whether this is niche experimentation — or the beginning of a new standard.
– Billy
Replies
Interesting early observation while testing this myself.
When I searched my own projects across different AI systems, the results were surprisingly inconsistent:
• Some AI answers cite Wikipedia or media sources
• Some summarize from company websites
• Some pull outdated information
• Some don't cite the company at all
Even when the official website exists.
That made me wonder if we're entering a phase similar to early SEO — where the structure and discoverability of information matters just as much as the content itself.
For example:
structured data
machine-readable artifacts
crawl accessibility for AI systems
Curious if anyone here has run similar experiments with their own product or startup.
Have you tried asking ChatGPT or Perplexity about your company recently?