For AI agents authoring these files

Why this page exists

The specifications define what a valid file contains. They do not define how an agent should go about producing one for a business it has just met — which facts it may infer, which it must ask for, and what to do when it cannot find an answer. That is what this page covers.

For what a valid file must contain, the specification documents remain the authority.

What to point an agent at

Four addresses, all public, no key required. If you are instructing an assistant yourself, the shortest useful prompt is: read https://discoveryfiles.ai/agent/authoring.md, then follow it.

curl -sS https://discoveryfiles.ai/agent/manifest.json

Where facts may come from

This is the part that decides whether the output is worth publishing. An agent can read a website and write eleven well-formed files in a minute; the question is whether the values in them are true.

The procedure

1. 01

   Check what already exists

   Fetch every path before writing anything. A site may already publish files that someone maintains deliberately. Overwriting a maintained brand.txt with your own inference is a regression, not an improvement.

   for f in llms.txt llm.txt llms-full.txt llms.html identity.json ai.json ai.txt brand.txt robots-ai.txt faq-ai.txt developer-ai.txt; do curl -sS -o /dev/null -w "%{http_code}  %{content_type}  $f\n" "https://example.com/$f"; done

   Stop condition If files already exist and look maintained, propose edits to the owner rather than replacing them.

2. 02

   Read the target site

   Gather what the owner has already published: services, locations, contacts, the language of the site, and the URLs worth linking. Note the exact wording the site uses about itself — matching it keeps the files consistent with the pages they describe.

3. 03

   Ask the owner for the gaps

   Present the “Ask the site owner” list as a short questionnaire. Ask for all of it at once rather than one question at a time. Make clear that unanswered items will be omitted, not guessed — that is a feature, and it usually prompts an answer.

   Stop condition Do not write identity.json without the registered business name and founding year confirmed, or explicitly waived.

4. 04

   Write the Layer 1 files first

   llms.txt and identity.json are Required — start there. Getting those right matters more than publishing all eleven. Start from the published templates rather than composing from memory: they carry the required section structure.

   curl -sS https://discoveryfiles.ai/agent/templates/llms.txt
   curl -sS https://discoveryfiles.ai/agent/templates/identity.json

5. 05

   Propagate the Core Identity block

   llms.txt is the source of truth. Write its Core Identity block from confirmed facts only, then repeat it word for word in every other file that carries it. Cross-file contradiction is the failure mode these specifications care most about.

6. 06

   Decide crawler policy with the owner, not for them

   robots-ai.txt and any AI training stance are business decisions, not technical ones. Explain the trade-off — blocking a training crawler is different from blocking a search crawler, and blocking the latter removes the business from that assistant’s answers — then implement what the owner chooses.

   Stop condition Never grant AI training permission without an explicit instruction from the owner.

7. 07

   Validate

   Run the repository validator over the finished files. Do not skip this because the output "looks right" — the validator catches exactly the errors that are invisible to reading.

   git clone --depth 1 https://github.com/GenerellAI/ai-discovery-files
   ./ai-discovery-files/scripts/validate.sh path/to/your/files/

8. 08

   Verify how they are served

   A valid file served with the wrong Content-Type, or behind a soft 404, is an unreadable file. Confirm the status line and content type after deploying — this is the only check that proves the work landed.

   curl -sSI https://example.com/llms.txt | grep -i '^HTTP\|^content-type'

9. 09

   Report what you omitted

   Tell the owner exactly which fields you left out and why, and which values came from them versus from their site. An omission they can fill in later is a good outcome; a plausible invention they never notice is the bad one.

The rules, compressed

Short enough to paste into a system prompt. This is the same list served in the manifest.

- Never invent a fact about the business. Omit instead.
- identity.json is authoritative. Every other file must agree with it.
- No superlatives, no prices, no competitor references, anywhere.
- Never grant AI training permission the owner has not explicitly given.
- Never write credentials, internal hostnames or staging URLs into developer-ai.txt.
- robots-ai.txt must not contradict robots.txt. robots.txt wins regardless.
- Every file needs a Last Updated date reflecting when it was actually written.
- Replace every bracketed placeholder, or delete the section containing it.
- Files go at the domain root. Language variants go at language paths, e.g. /fr/llms.txt.
- Report omissions to the owner rather than filling them silently.

Raw template URLs

Fetch a template rather than composing a file from memory. Every value that must be supplied is wrapped in [SQUARE BRACKETS], which also makes it trivial to check afterwards whether any were left behind.

grep -l "\[" llms.txt ai.txt brand.txt faq-ai.txt developer-ai.txt robots-ai.txt