updated docker-compose persistence

[katmai]

Background

Persistence improvements in docker-compose.yml for the workspace, ai_settings and redis

Changes

• the app writes to: /home/appuser/auto_gpt_workspace and not to /app/auto_gpt_workspace.
• included mount for ai_settings.yml so we don't have to recreate the settings on subsequent runs.
• added persistence for redis data
• added redis-stack.conf which configures regular 'aof' writes for backup purposes.

Documentation

Test Plan

PR Quality Checklist

• [X] My pull request is atomic and focuses on a single change - data persistence.
• [X] I have thoroughly tested my changes with multiple different prompts.
• [ ] I have considered potential risks and mitigations for my changes.
• [X] I have documented my changes clearly and comprehensively.
• [ ] I have not snuck in any "extra" small tweaks changes

[katmai]

i'm having issues with pytest not allowing me to commit. can anyone help me out on this? i disabled it temporarily so i can push the fixes.

pytest-check.............................................................Failed

• hook id: pytest-check
• exit code: 4

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --cov=autogpt --without-integration --without-slow-integration inifile: None rootdir: /home/katmai/AI/Auto-GPT

[crushingCodes]

i'm having issues with pytest not allowing me to commit. can anyone help me out on this? i disabled it temporarily so i can push the fixes.

pytest-check.............................................................Failed

• hook id: pytest-check
• exit code: 4

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --cov=autogpt --without-integration --without-slow-integration inifile: None rootdir: /home/katmai/AI/Auto-GPT

@katmai Did you check the section about pre commit hooks? https://github.com/Significant-Gravitas/Auto-GPT/blob/master/CONTRIBUTING.md#pre-commit-hooks

[katmai]

i'm having issues with pytest not allowing me to commit. can anyone help me out on this? i disabled it temporarily so i can push the fixes. pytest-check.............................................................Failed

• hook id: pytest-check
• exit code: 4

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --cov=autogpt --without-integration --without-slow-integration inifile: None rootdir: /home/katmai/AI/Auto-GPT

@katmai Did you check the section about pre commit hooks? https://github.com/Significant-Gravitas/Auto-GPT/blob/master/CONTRIBUTING.md#pre-commit-hooks

i did but i don't see anything relevant to the issue i'm seeing. i installed the hooks.

[katmai]

i'm having issues with pytest not allowing me to commit. can anyone help me out on this? i disabled it temporarily so i can push the fixes. pytest-check.............................................................Failed

• hook id: pytest-check
• exit code: 4

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: --cov=autogpt --without-integration --without-slow-integration inifile: None rootdir: /home/katmai/AI/Auto-GPT

@katmai Did you check the section about pre commit hooks? https://github.com/Significant-Gravitas/Auto-GPT/blob/master/CONTRIBUTING.md#pre-commit-hooks

i was using python 3.8 for some reason. lord hesus ....

[katmai]

fixed everything. sorry about that. i sort of messed up my env for a little while. all cleaned up.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[montanaflynn]

Looks like this is missing the fix for auto_gpt_workspace being put in the ./autogpt/auto_gpt_workspace directory.

I'd prefer if it just used the ./auto_gpt_workspace next to the ./ai_settings.yaml as I expected.

Speaking of ai_settings.yaml, I didn't have to recreate it on subsequent runs.

[codecov[bot]]

Codecov Report

Patch coverage has no change and project coverage change: -16.25 :warning:

Comparison is base (b8478a9) 57.19% compared to head (5564e1f) 40.94%.

:exclamation: Current head 5564e1f differs from pull request most recent head f9a5023. Consider uploading reports for the commit f9a5023 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3015       +/-   ##
===========================================
- Coverage   57.19%   40.94%   -16.25%     
===========================================
  Files          68       64        -4     
  Lines        3079     2950      -129     
  Branches      516      499       -17     
===========================================
- Hits         1761     1208      -553     
- Misses       1180     1683      +503     
+ Partials      138       59       -79     

see 43 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[katmai]

Thanks for the above fixes, one more thing i noticed while using this config is i get git changes, from redis data, could you add the following to .gitignore please redis-data/*

done

[Pwuts]

A question though: does Redis need to be perpetually persistent? The entire idea of the current setup is that Redis stays up as long as needed, and by default it is wiped at the beginning of every run. So in what use case would you need this level of persistence?

If persistence is an actual problem, please link related issues.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		Apr 28, 2023 10:08pm

[katmai]

Please revert the change to the redis service name. Other than that, looks good! :)

name changed back to redis

[katmai]

A question though: does Redis need to be perpetually persistent? The entire idea of the current setup is that Redis stays up as long as needed, and by default it is wiped at the beginning of every run. So in what use case would you need this level of persistence?

If persistence is an actual problem, please link related issues.

why wouldn't redis be persistent though, and why would we want to wipe the memory on every run? isn't the goal to somehow in the future optimize the usage of the data stored in persistent memory on subsequent runs, so that we can reuse the information we've already learned ? wouldn't that also be giving us an individual character of the AI instance that we run locally ?

[Pwuts]

why wouldn't redis be persistent though, and why would we want to wipe the memory on every run?

Because this project is far from refined, and errors often carry over to subsequent runs if memory is persisted. Also, when changing an agent from one task to another, the memory should be wiped or it will cause confusion.

For current use cases, the data persists as long as the Redis container exists with WIPE_REDIS_ON_START=False. This is enough for most users.

isn't the goal to somehow in the future optimize the usage of the data stored in persistent memory on subsequent runs, so that we can reuse the information we've already learned ?

In a way, yes. But that is a use case for permanent memory, not for working memory (cache) as it is implemented now.

[katmai]

isn't the goal to somehow in the future optimize the usage of the data stored in persistent memory on subsequent runs, so that we can reuse the information we've already learned ?

In a way, yes. But that is a use case for permanent memory, not for working memory (cache) as it is implemented now.

what's considered permanent memory ?

[Pwuts]

There was a (non-working draft) module in the repo before that tried to implement permanent memory using an SQL database. That is permanent memory. Pinecone, Redis etc are all currently intended as volatile medium-term memory to hold context for the task that is being executed.

[katmai]

so why do we present these things as THOUGHTS and REASONING then?

[katmai]

Please revert the change to the redis service name. Other than that, looks good! :)

this was done btw.

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[katmai]

i'll close this. not really needed right now.

[anonhostpi]

@katmai don't get too discouraged. The re-arch is on a hot streak right now, the Plugin, Command, Planner, and Workspace structures of the API have already been reworked. Definitely a good idea though.

I would plan on reopening your PR or duplicating it after the rework is done.