apify
Improve markdown structure of docs llms.txt & Actor detail
Epic for improving the current state of the llms.txt file (https://docs.apify.com/llms.txt) and the Actor detail markdown version (e.g., https://apify.com/apify/web-scraper.md)
Hey team, please keep me posted about the progress, I'll be happy to test the new version
One more feedback:
Let's add to the very top of https://docs.apify.com/llms.txt this text, to make this info easy to find for everyone. And add some basic text there as well, to introduce the Apify platform and the docs.
The full Markdown export of the docs is available at: https://docs.apify.com/llms-full.txt
Guys, what is the status for this project? Seems there is no progress over the past 2 weeks
@jancurn We've merged continuation few hours back: https://github.com/apify/apify-docs/pull/1796, also created ticket for fixing description as it doesn't seem llms issue, but rather general Docusaurus one: https://github.com/apify/apify-docs/issues/1870
Hey guys, please what is the status of this?
Hey @jancurn, we've prioritized and fixed the main issues:
- We made the llms.txt structure hierarchical
- We improved the llms.txt for Crawlee
- We fixed the description issues for the links
We're planning to work on the other details in the upcoming sprints
@jbartadev thanks for the info and for the progress. There are still low-hanging fixes that I believe we could do:
See https://github.com/jancurn/test-llm-txt/blob/main/README.md how the llms.txt file renders.
Please let's add to the top of llms.txt this text:
The entire content of Apify documentation is available in a single Markdown file at https://docs.apify.com/llms-full.txt
The word "Clients" leaks into the description text.
Also some formatting leaks into the descriptions. I think we could just remove all Markdown formatting from this and convert it to plain text, it will be better in all these cases.
https://docs.apify.com/cli.md file doesn't exist, even though https://docs.apify.com/cli does - let's be consistent.
The alternate tag must contain absolute URL.
Thing to consider:
AI Agents usually doesn't send text/html in Accept header, but text/markdown or text/plain instead. We can try to serve plain/markdown content right away there.
see https://x.com/bunjavascript/status/1971934734940098971 https://x.com/NathanFlurry/status/1972427441898717607
EDIT: Related Slack conversation: https://apify.slack.com/archives/CQ96RHG2U/p1759086275925279
AI Agents usually doesn't send text/html in Accept header, but text/markdown or text/plain instead. We can try to serve plain/markdown content right away there.
FYI I am already working on that part.
Another observation - the order of items in llms.txt doesn't correspond to the order of pages in the docs menu
Another feedback - mailto links are malformatted. E.g. see https://docs.apify.com/platform/integrations/actors/integration-ready-actors
@jancurn not sure what's wrong with them, subject query param is valid one in mailto: https://stackoverflow.com/a/4782084/7125721
It should end up as [contact support](mailto:...) instead of just the mailto link itslef.