Add llms.txt

[chrisvxd]

Description

Add llms.txt for all docs

Considerations

• How can we avoid duplication?
• How do we handle open-source vs non-open-source docs?

[FedericoBonel]

It would be great if we could automate this somehow.

One thing I think we should consider is that, while we write everything in MDX, some content is not only markdown and comes after rendering the components to HTML, so we might want to either:

• Use the HTML for parsing and generating llms.txt
• Use the mdx files, parse them to transform to markdown by trimming or extracting docs from components, and assume that's good enough
• Manually create them with every new piece of content

There might be a converter from MDX to LLMs.txt library, I did see nextra has a feature request for it, but it hasn't been implemented yet.

Anthropic uses two files: llms.txt and llms-full.txt, the first one is an index/ToC kind of thing, the second one is the full docs in markdown.