[Feature] Conversation Summary Mode: Track Assistant Responses

[scriptogre]

Background

This pull request introduces the "Conversation Summary Mode," which generates concise GPT-generated summaries for each step and a final summary upon exit. It enables users to track the assistant's responses and identify deviations, ultimately improving its functionality.

To activate the mode, run main.py with -cs or --conversation-summary:

python scripts/main.py --conversation-summary

Changes

• Introduced Conversation Summary Mode, generating concise step summaries and a final summary upon exit

• Refactored main.py to support the new mode while maintaining existing functionality

• Created Summary and SummaryUtils classes to manage summarization features and handle large text

• Added test coverage for all new methods and classes

• Updated .env file to accept STEP_SUMMARIZATION_PROMPT and FINAL_SUMMARIZATION_PROMPT variables, allowing users to customize prompts and improve summarization results

Documentation

The new features are accompanied by comprehensive docstrings for each newly added method, code comments where needed, and an updated README.md section explaining how to set up and use this mode. All changes are modular, and the existing code has been minimally altered to adhere to the Separation of Concerns principle.

Test Plan

Tests cover the initialization of a Summary instance, incrementation of the step counter, printing of step and final summaries to the console, appending of step summaries and results to the summary file, formatting of the final summary prompt, and extraction of the step summary from the assistant's reply.

PR Quality Checklist

• [?] My pull request is atomic and focuses on a single change It's not really a single change, but it's a very modular change. I tried my best not to touch the existing code too much but keep all the new functionality inside the conversation_summary library for applying the Separation of Concern principle.

• [✓] I have thoroughly tested my changes with multiple different prompts.

• [✓] I have considered potential risks and mitigations for my changes.

• [✓] I have documented my changes clearly and comprehensively.

• [✓] I have not snuck in any "extra" small tweaks changes

Potential risks and mitigations:

Risk: Summaries may not always be coherent or helpful due to GPT's inherent limitations. Mitigation: Carefully monitor the output to ensure the feature is working as expected and adjust the prompts and code as needed to improve the quality of summaries.

Risk: Large texts or numerous steps may cause the script to take a long time to generate summaries or hit API rate limits. Mitigation: Implement chunking to handle large texts and limit the number of steps during testing. Additionally, monitor API usage to avoid unexpected costs or throttling.

Risk: Unexpected bugs or issues may arise when the new mode is enabled or when working with the modified code. Mitigation: Thoroughly test the changes and monitor the output during usage. Address any issues as they arise and consider adding more tests to cover edge cases.

[blankster]

@scriptogre Thank you so much for tackling exactly what I was looking for. In the current state, I find the meta-level action strategy alone fascinating - to capture it in an automated descriptive way is extremely nice.

It is extremely exciting to put Auto-GPT on a question where it works as a researcher. If you know your stuff and have put hundreds if not thousands of hours into it, it's super exciting to see how Auto-GPT eventually (when it's not busting its balls over a JSON error or losing its memory again) tends to act similarly, so you can say that 95% of its actions are really useful and interesting - and 1000x faster.

The topic of reporting/extraction of knowledge and automatic knowledge discovery and summarization/structuring I feel has huge potential and is underrated.

PS: Very off-topic, but a "secret desire" that would be pretty nice, if it could be fed (embeddings?) with available information, it starts to validate and extend. It doesn't need to fail trying to download scientific papers, if there is a pretty nice Zotero library with structured and rich content available. Maybe also interconnecting it with Galactica... That would be my ultimate dream.

[scriptogre]

Wow, feels good to know my work proves to be of use to other users. My intention is to develop a succinct summarization that can be fed back to the AI model, ensuring it remains focused and eliminating the need to reset it after each terminal session. By maintaining a record of past actions and objectives, the AI can seamlessly resume its tasks. The ultimate goal is to generate an efficient, multi-layered summary that encapsulates the AI's progress and ongoing tasks, effectively communicating its status in a clear and concise manner.

[scriptogre]

@Torantulino What do you think? Any issues with the PR? It's a pretty nice functionality for making the auto-gpt a more robust tool. I hope you consider merging it ☺️

[nponeccop]

@scriptogre There are conflicts now

[scriptogre]

@scriptogre There are conflicts now

Yup! Seen those. Was about to fix them today but didn't get the chance. Will fix them tomorrow.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[Boostrix]

This would also seem like something that would be best done using dynamic prompting as per #3937

[Pwuts]

Interesting concept! It seems somewhat similar to the rolling summary that is currently being used for the agent's message history. If anyone is interested in re-evaluating the value of this PR and comparing it to the current implementation of MessageHistory that would be grand. For now, this PR has extensive merge conflicts so I'll close, with the hope that this will be looked into further.