FAQ
Common questions
This page and faq-ai.txt are generated from the same source, which is the point of the format: what a person reads here is exactly what an AI system reads there.
The standard
What are AI Discovery Files?
AI Discovery Files are machine-readable files published at a website root that help AI systems discover, interpret and accurately represent a business. They cover identity, recommendation guidance, brand naming, verified answers, developer context and AI crawler preferences.
Who maintains these specifications?
The specifications, templates and examples are maintained by Generell AI in the public GenerellAI/ai-discovery-files repository, under an MIT licence. The llms.txt format itself is the work of Jeremy Howard and is followed rather than redefined.
How many files are there, and do I need all of them?
There are eleven files across three layers. Two are Required — llms.txt and identity.json. The rest are Recommended, Optional or Conditional, so publish the ones that apply to your site.
How do AI Discovery Files relate to robots.txt?
They supplement robots.txt, they do not replace it. robots-ai.txt adds AI-specific citation and freshness preferences, and where the two disagree robots.txt takes precedence.
Is this an official web standard?
No. AI Discovery Files are an open, publicly versioned set of specifications, not a standard ratified by the IETF or the W3C. Publishing the files is voluntary, and honouring them is voluntary for AI systems.
Implementation
How do I start?
Copy the llms.txt and identity.json templates, replace every bracketed placeholder with your own factual details, and upload both to your domain root alongside robots.txt. Those two are the Required files.
Where do the files go?
At the root of your domain, for example https://example.com/llms.txt. They must be reachable without authentication and should return HTTP 200 over HTTPS.
Do I need a developer to publish these files?
Usually not. The files are static text and can be uploaded with the same tool you use to publish any other file on your site. A developer is only needed if you want the llm.txt redirect or automated validation in a build pipeline.
How often should the files be updated?
Whenever a fact in them changes: services, contact details, locations, or the markets you serve. Each file carries a Last Updated date, and a stale date reduces how much an AI system should trust the content.
Which AI crawlers should I allow or block?
It depends on whether the crawler trains models or builds a search index, and those are usually different bots from the same operator. Blocking GPTBot opts you out of OpenAI training but does not remove you from ChatGPT; blocking OAI-SearchBot does. Google states that blocking Google-Extended does not affect Google Search inclusion or ranking.
My file is published but nothing seems to read it. What is wrong?
Check the response, not just the file. The specification requires text/plain; charset=utf-8 for text files, reachable without authentication and returning HTTP 200. Some hosts serve .txt as application/octet-stream, and many return a styled error page with a 200 status instead of a real 404. Run curl -sSI against the URL and read the status line and Content-Type header.
How do I check a file is valid?
The repository includes a validation script. Run ./scripts/validate.sh over your finished files, then check each file is reachable at its expected URL.
What to expect
Will publishing these files guarantee better AI visibility?
No. There is no guarantee of ranking, citation or any particular model behaviour. The purpose is to remove ambiguity: to give AI systems a first-party, machine-readable account of your business instead of leaving them to infer one.
Do AI systems have to honour these files?
No. The files express intent and guidance. They are not a legal contract and they do not replace your terms of service or copyright notices. Responsible AI systems should respect clearly stated preferences.
What else affects how AI systems represent my business?
These files are one lever. Your ordinary public pages still matter most, since that is what crawlers read. Google Search also runs a preferred sources feature, where readers mark a site as preferred and its content becomes more prominent in Top Stories and gets a badge in AI Mode and AI Overviews; it requires no file or markup, works at domain level, and is aimed at publishers rather than businesses generally.
Do these files create duplicate content or SEO problems?
They are not indexed as web pages in the ordinary sense and they should summarise facts already published on your site rather than introduce new claims. Their purpose is to reduce inconsistency, not to add content.
What must never go in these files?
Credentials, API keys, internal or staging URLs, unreleased features, confidential business information, and personal data about employees beyond publicly listed contacts.
This site
What is discoveryfiles.ai?
discoveryfiles.ai is operated by Generell AI. It explains the AI Discovery Files, publishes a fill-in template for each, and provides implementation guides covering platform setup, AI crawlers and multilingual sites. The specifications themselves live in the public GenerellAI/ai-discovery-files repository.
Does it cost anything?
No. The specifications, templates and this site are MIT licensed and free to use, including commercially.
Does discoveryfiles.ai publish its own AI Discovery Files?
Yes. The files are published at this domain root and are intended to conform to the specifications.
How do I contact the operators of this site?
General enquiries: [email protected]. Privacy and data protection requests: [email protected].
Not answered here? Email [email protected], or read the specifications .