autogen icon indicating copy to clipboard operation
autogen copied to clipboard

Add role to reflection with llm

Open MarianoMolina opened this issue 1 year ago • 9 comments

Use 'role' in summary_args to use as the role string of the prompt message used to summarize.

Why are these changes needed?

When creating a GroupChat summary, the prompt 'role' is hardcoded as "system". Idea is to enable the initiate_chat() flow to define this by declaring it in system_args. Issue is that there is already a 'role' argument that is passed: (From the docstrings)

"role": role of the message, can be "assistant", "user", "function". This field is only needed to distinguish between "function" or "assistant"/"user".

However, I don't see anywhere in the flow that this value is retrieved or used for anything. I'm not entirely sure what the intention was (potentially to distinguish function calls from conversations, but the flow makes that decision irrespective of the 'role' value), so we can either (1) repurpuse this value to use as the role string to pass in the message dict or (2) add a different argument (called 'summary_role' or something) to avoid potential issues.

Given that, to the best of my knowledge, the old 'role' value wasn't being used, I think the best approach is 1.

With these changes, the argument can still be used to distinguish between function and assistant, but now has a 'default' value of 'system', and is used as the string value for the 'role' attribute in the summary prompt message.

  • No doc changes needed

Related issue number

Closes #2492

Checks

  • [] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
  • [x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
  • [ ] I've made sure all auto checks have passed.

MarianoMolina avatar Apr 26 '24 16:04 MarianoMolina

Added a test on test_groupchat.

MarianoMolina avatar Apr 27 '24 16:04 MarianoMolina

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
10404662 Triggered Generic CLI Secret eff19acf1365e34fe17d9ac0939666f32b3ceda5 .github/workflows/dotnet-release.yml View secret
10404662 Triggered Generic CLI Secret e973ac38ea4f7b36687ea03aa44f770d7e2ddcac .github/workflows/dotnet-release.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

gitguardian[bot] avatar Apr 29 '24 15:04 gitguardian[bot]

https://github.com/microsoft/autogen/actions/runs/8881467046/job/24383882709?pr=2527 The code formatting error can be fixed by pre-commit.

https://github.com/microsoft/autogen/actions/runs/8881467028/job/24383887017?pr=2527 The build error is due to the use of "OAI_CONFIG_LIST" in the added test. Could you use a mocked config list for this test which doesn't require real openai keys?

sonichi avatar Apr 29 '24 16:04 sonichi

The changes are fine. Please follow @sonichi 's comments. Thanks.

ekzhu avatar Apr 30 '24 17:04 ekzhu

Fixed both comments.

MarianoMolina avatar Apr 30 '24 18:04 MarianoMolina

The code formatting error can be fixed with pre-commit.

sonichi avatar May 02 '24 21:05 sonichi

I installed pre-commit, but since I am not using WSL, I had to uninstall to be able to commit.

MarianoMolina avatar May 03 '24 17:05 MarianoMolina

Please fix the build error.

sonichi avatar May 04 '24 21:05 sonichi

@microsoft-github-policy-service agree

MarianoMolina avatar May 07 '24 22:05 MarianoMolina

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 22.84%. Comparing base (372ac1e) to head (b23b4ba). Report is 17 commits behind head on main.

Files Patch % Lines
autogen/agentchat/conversable_agent.py 0.00% 6 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2527       +/-   ##
===========================================
- Coverage   33.11%   22.84%   -10.27%     
===========================================
  Files          86       86               
  Lines        9108     9243      +135     
  Branches     1938     2131      +193     
===========================================
- Hits         3016     2112      -904     
- Misses       5837     6888     +1051     
+ Partials      255      243       -12     
Flag Coverage Δ
unittests 22.78% <0.00%> (-10.33%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar May 13 '24 10:05 codecov-commenter

Please fix the CI error on formatting.

You can install pre-commit so this will be checked before commit. https://microsoft.github.io/autogen/docs/contributor-guide/pre-commit

ekzhu avatar May 13 '24 10:05 ekzhu

Not sure how or why, but seems the issue is with the test_groupchat file, which got an update in main that commented all functions and created this issue. This merge from main seems to be the cause this issue was present in this branch: https://github.com/microsoft/autogen/pull/2527/commits/b23b4bafbac51f6e32ff722e9ec4ebeea766b6dc

MarianoMolina avatar May 13 '24 22:05 MarianoMolina