[feat] Aim integration

[tamohannes]

🚀 Feature

Background

Integrating with Aim would be fantastic, as it would provide a convenient method for tracking metadata and storing experiment histories in an intuitive and visually pleasing manner. To enhance experiment reproducibility and provide a unified, aesthetically pleasing user interface for exploring experiments and their corresponding input/outputs, it is desirable to furnish users with a fast and a solid tool. Aim facilitates gaining deeper insights, thus enhancing the quality of experimentation.

Changes

The following changes are made in the codebase

• Created AimCallback to track input prompts and agent replies, in a separate aim.py file.
• Added cli option --aim-repo to give a specific Aim repo path if needed.
• Passed an instance of AimCallback: aim_callback to track the logs in start_interaction_loop method

Documentation

I've included in-code comments to make it easy for the reviewer to understand.

Test Plan

I executed Auto-GPT with some tasks and thoroughly explored the Aim UI during the response period of the interaction loop.

PR Quality Checklist

• [ ] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [ ] I have not snuck in any "extra" small tweaks changes

There is no need for any specific tests. By running Auto-GPT and examining Aim explorer, you will have all the information you need.

[ntindle]

Hey! Would love some more context about aim in the PR and why it’s useful

[tamohannes]

Hello @ntindle, thank you for your prompt response. Aim is an open-source AI metadata tracker that is designed to be user-friendly and highly efficient. With Aim, you can effortlessly log all your AI metadata, and utilize its intuitive UI to observe and compare them. Additionally, Aim comes equipped with an SDK that enables you to query metadata programmatically.

By using Aim, you can ensure the reproducibility of your experiments, as it helps you to keep track of all the generated responses. Overall, Aim is a powerful tool that simplifies AI metadata management and enhances the efficiency of your workflow.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[ntindle]

@merwanehamadi

[tamohannes]

Hey @ntindle, it seems like there are numerous unrelated files present, which is making the repository look messy. Let me open another PR to keep things clean. By the way, I wanted to check if you've completed the phase of re-architecting the core. Is it relatively stable now, so I can proceed with opening the PR?

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		May 26, 2023 3:42pm

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[tamohannes]

@ntindle, nevermind. don't need to open a new PR. have successfully resolved the conflicts. :)

[ntindle]

Ref: https://github.com/aimhubio/aim

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[vercel[bot]]

Deployment failed with the following error:

Resource is limited - try again in 22 minutes (more than 100, code: "api-deployments-free-per-day").

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[vercel[bot]]

Deployment failed with the following error:

Resource is limited - try again in 11 minutes (more than 100, code: "api-deployments-free-per-day").

[vercel[bot]]

Deployment failed with the following error:

Resource is limited - try again in 4 minutes (more than 100, code: "api-deployments-free-per-day").

[codecov[bot]]

Codecov Report

Patch coverage: 20.27% and project coverage change: -0.98 :warning:

Comparison is base (acfd966) 67.71% compared to head (325f402) 66.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3101      +/-   ##
==========================================
- Coverage   67.71%   66.74%   -0.98%     
==========================================
  Files          72       73       +1     
  Lines        3528     3602      +74     
  Branches      557      570      +13     
==========================================
+ Hits         2389     2404      +15     
- Misses        955     1012      +57     
- Partials      184      186       +2     

Impacted Files	Coverage Δ
autogpt/cli.py	0.00% <0.00%> (ø)
autogpt/main.py	0.00% <0.00%> (ø)
autogpt/aim.py	20.63% <20.63%> (ø)
autogpt/agent/agent.py	58.23% <28.57%> (-1.28%)	:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[tamohannes]

@ntindle, it appears that all of the workflows have been completed successfully (not quite sure what to do with "codecov/patch — 20.27% of diff hit (target 80.00%)"). Please let me know if there's anything else I can do to contribute towards merging this. 😊

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[tamohannes]

Resolves #4842