AutoGPT icon indicating copy to clipboard operation
AutoGPT copied to clipboard

Large file fix

Open pabstaaron opened this issue 2 years ago • 3 comments

Fixes issues related to reading large amounts of text from input files.

Checks the token count of an input text before reading it into memory. If the file contains too many tokens, returns a system error stating that the input must be summarized.

Also adds a command to summarize the contents of a file in a similar manner to web pages, which seems an essential command and enables Auto to complete workflows involving large file inputs.

pabstaaron avatar Apr 30 '23 17:04 pabstaaron

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
docs ⬜️ Ignored (Inspect) May 1, 2023 11:31pm

vercel[bot] avatar Apr 30 '23 17:04 vercel[bot]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions[bot] avatar May 01 '23 18:05 github-actions[bot]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

github-actions[bot] avatar May 01 '23 23:05 github-actions[bot]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

p-i- avatar May 05 '23 00:05 p-i-

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions[bot] avatar May 21 '23 20:05 github-actions[bot]

Summarizing files when reading them is now built-in (#4208). Nonetheless, I like the idea of letting the LLM choose between reading literal content or just a summary of a file. This is going in my notes, maybe we can do something with it while working on retrieval augmentation (#3536).

Thanks for submitting!

Pwuts avatar Jun 07 '23 09:06 Pwuts