Skip to main content

Web and LLM Discoverability

Files and well-known URLs that help LLMs discover docs, pricing, and APIs.

How To Evaluate Entries​

Each entry in this family should make the following points clear:

  • What the convention is.
  • Where the file, URL, or protocol surface normally lives.
  • When a team should use it.
  • Adoption evidence from a public spec, canonical docs, or active ecosystem use.
  • Which example illustrates the convention, if one exists.
  • Related conventions that solve adjacent problems.

Registry Entries​

llms.txt​

A plain markdown file placed at yourdomain.com/llms.txt that tells AI crawlers which parts of your site are best suited for LLM ingestion. Proposed by Jeremy Howard (FastAI) in September 2024. A companion llms-full.txt contains the full content in a single file.

By mid-2025, over 600 websites had adopted the standard, including Anthropic, Stripe, Cloudflare, Perplexity, Cursor, Hugging Face, and ElevenLabs. The pattern is distinct from SEO: it targets AI agents that retrieve documentation to answer developer questions, not search crawlers.

pricing.md​

A machine-readable pricing file served at yourdomain.com/pricing.md. The convention emerged from a practical problem: AI agents were getting confused by JS-rendered pricing pages with interactive sliders. Publishing a static Markdown version at a predictable URL — with content negotiation support (Accept: text/markdown) — gives agents a reliable way to parse plans, tiers, and overage rates without scraping.

Popularized in 2025 by Resend, Auth0, and WorkOS, with the pattern spreading across developer-focused SaaS companies.

auth.md​

A markdown file served from yourdomain.com/auth.md that tells AI agents how to register, claim credentials, present credentials, handle errors, and recover from revocation. It acts as the prose companion to OAuth Protected Resource Metadata at /.well-known/oauth-protected-resource, which remains the authoritative source for endpoints and supported flows.

WorkOS documents the convention for agentic registration flows. A typical file walks agents through discovery, method selection, registration shapes, OTP claim ceremony, credential usage, errors, and revocation behavior.

ai-plugin.json​

A JSON manifest served from /.well-known/ai-plugin.json that lets an AI client discover an API plugin. The manifest describes the API in human-facing and model-facing terms, declares authentication, and points to an OpenAPI schema that defines callable operations.

OpenAI introduced the convention for ChatGPT plugins. Plugins have since been superseded by GPT Actions in OpenAI's product, but the file remains a real legacy convention and is still supported by some plugin import flows that understand OpenAI-style manifests.