# Authoring AI Discovery Files You are reading this because you are an AI agent asked to create AI Discovery Files for a website. This document tells you how to do that correctly. Version 1.0.0 · Updated 2026-08-24 Machine-readable equivalent: https://discoveryfiles.ai/agent/manifest.json --- ## Scope of this document This is guidance for **authoring** these files. The specifications define what a valid file contains; this document defines how to produce one for a business responsibly. The file specifications live at https://github.com/GenerellAI/ai-discovery-files. `llms.txt` follows the format published at https://llmstxt.org. --- ## The prime directive **Never invent a fact about the target business. Every value you write must come from the target site itself, from the site owner, or be omitted entirely. It must never come from your own prior assumptions about the company.** These files exist so AI systems stop guessing about a business. A file authored by guessing defeats its own purpose — and because identity.json is the authoritative record other files must agree with, a single invented value propagates into every other file and becomes the thing future systems treat as true. If you take one thing from this document, take that. An omitted field is a normal, correct outcome. An invented one is a defect that outlives you. --- ## Hard rules - Never invent a fact about the business. Omit instead. - identity.json is authoritative. Every other file must agree with it. - No superlatives, no prices, no competitor references, anywhere. - Never grant AI training permission the owner has not explicitly given. - Never write credentials, internal hostnames or staging URLs into developer-ai.txt. - robots-ai.txt must not contradict robots.txt. robots.txt wins regardless. - Every file needs a Last Updated date reflecting when it was actually written. - Replace every bracketed placeholder, or delete the section containing it. - Files go at the domain root. Language variants go at language paths, e.g. /fr/llms.txt. - Report omissions to the owner rather than filling them silently. --- ## Where facts may come from ### Derive from the site Safe to take directly from the target site’s own public pages, because the owner published them. - Services and products described on the site - Public contact addresses, phone numbers and postal addresses - Office or location names listed publicly - Social profile URLs linked from the site - Existing page URLs for the link sections of llms.txt - The site’s primary language ### Ask the site owner Cannot be reliably scraped and must not be guessed. If the owner cannot answer, omit the field rather than fill it. - Registered legal entity name, if it differs from the trading name - Company registration number and VAT number - Founding date - Services the business explicitly does NOT offer - Markets or regions explicitly NOT served - Whether AI training on their content is permitted - Brand forms that must never be used — retired names, wrong abbreviations - The contact address for AI-related enquiries ### Never write Prohibited by the specifications, or too risky for an agent to decide unilaterally. - Any superlative — "leading", "world-class", "best-in-class", "innovative" - Prices, which change faster than files are updated - Headcount, revenue or funding figures not stated on the site - Competitor names or comparative claims - Testimonials, reviews or quotes you cannot source - Credentials, API keys, internal hostnames or staging URLs - A permissive AI training grant the owner has not explicitly given --- ## Procedure ### 1. Check what already exists Fetch every path before writing anything. A site may already publish files that someone maintains deliberately. Overwriting a maintained brand.txt with your own inference is a regression, not an improvement. ```bash for f in llms.txt llm.txt llms-full.txt llms.html identity.json ai.json ai.txt brand.txt robots-ai.txt faq-ai.txt developer-ai.txt; do curl -sS -o /dev/null -w "%{http_code} %{content_type} $f\n" "https://example.com/$f"; done ``` **Stop condition:** If files already exist and look maintained, propose edits to the owner rather than replacing them. ### 2. Read the target site Gather what the owner has already published: services, locations, contacts, the language of the site, and the URLs worth linking. Note the exact wording the site uses about itself — matching it keeps the files consistent with the pages they describe. ### 3. Ask the owner for the gaps Present the “Ask the site owner” list as a short questionnaire. Ask for all of it at once rather than one question at a time. Make clear that unanswered items will be omitted, not guessed — that is a feature, and it usually prompts an answer. **Stop condition:** Do not write identity.json without the registered business name and founding year confirmed, or explicitly waived. ### 4. Write the Layer 1 files first llms.txt and identity.json are Required — start there. Getting those right matters more than publishing all eleven. Start from the published templates rather than composing from memory: they carry the required section structure. ```bash curl -sS https://discoveryfiles.ai/agent/templates/llms.txt curl -sS https://discoveryfiles.ai/agent/templates/identity.json ``` ### 5. Propagate the Core Identity block llms.txt is the source of truth. Write its Core Identity block from confirmed facts only, then repeat it word for word in every other file that carries it. Cross-file contradiction is the failure mode these specifications care most about. ### 6. Decide crawler policy with the owner, not for them robots-ai.txt and any AI training stance are business decisions, not technical ones. Explain the trade-off — blocking a training crawler is different from blocking a search crawler, and blocking the latter removes the business from that assistant’s answers — then implement what the owner chooses. **Stop condition:** Never grant AI training permission without an explicit instruction from the owner. ### 7. Validate Run the repository validator over the finished files. Do not skip this because the output "looks right" — the validator catches exactly the errors that are invisible to reading. ```bash git clone --depth 1 https://github.com/GenerellAI/ai-discovery-files ./ai-discovery-files/scripts/validate.sh path/to/your/files/ ``` ### 8. Verify how they are served A valid file served with the wrong Content-Type, or behind a soft 404, is an unreadable file. Confirm the status line and content type after deploying — this is the only check that proves the work landed. ```bash curl -sSI https://example.com/llms.txt | grep -i '^HTTP\|^content-type' ``` ### 9. Report what you omitted Tell the owner exactly which fields you left out and why, and which values came from them versus from their site. An omission they can fill in later is a good outcome; a plausible invention they never notice is the bad one. --- ## The files Publish every file at the domain root, never in a subdirectory. | File | Layer | Priority | Template | |---|---|---|---| | `llms.txt` | Core Identity | Required | https://discoveryfiles.ai/agent/templates/llms.txt | | `llm.txt` | Core Identity | Required | https://discoveryfiles.ai/agent/templates/llm.txt | | `llms-full.txt` | Core Identity | Conditional | https://discoveryfiles.ai/agent/templates/llms-full.txt | | `llms.html` | Core Identity | Recommended | https://discoveryfiles.ai/agent/templates/llms.html | | `identity.json` | Core Identity | Required | https://discoveryfiles.ai/agent/templates/identity.json | | `ai.json` | Core Identity | Recommended | https://discoveryfiles.ai/agent/templates/ai.json | | `ai.txt` | Guidance & Control | Recommended | https://discoveryfiles.ai/agent/templates/ai.txt | | `brand.txt` | Guidance & Control | Recommended | https://discoveryfiles.ai/agent/templates/brand.txt | | `robots-ai.txt` | Guidance & Control | Optional | https://discoveryfiles.ai/agent/templates/robots-ai.txt | | `faq-ai.txt` | Enhancement | Recommended | https://discoveryfiles.ai/agent/templates/faq-ai.txt | | `developer-ai.txt` | Enhancement | Conditional | https://discoveryfiles.ai/agent/templates/developer-ai.txt | Start with the Required files in Layer 1. Publishing those correctly beats publishing all eleven carelessly. Fetch a template rather than composing a file from memory — the templates carry the required section structure, and every value you must supply is wrapped in `[SQUARE BRACKETS]`. --- ## Validation ```bash git clone --depth 1 https://github.com/GenerellAI/ai-discovery-files ./ai-discovery-files/scripts/validate.sh path/to/your/files/ ``` Then confirm how the files are actually served: ```bash curl -sSI https://example.com/llms.txt | grep -i '^HTTP\|^content-type' ``` Expect `200` and `text/plain; charset=utf-8`. A styled error page returned with a 200 status reads to a consumer as a valid file containing your error page. --- ## Consistency The Core Identity block in `llms.txt` is the source of truth. Repeat it word for word in every file that carries it. Where files disagree, `llms.txt` is correct and the others need fixing. --- ## If something here is wrong Report it to info@discoveryfiles.ai. Platform behaviour and crawler names change; this document is maintained, not archived.