Significant-Gravitas
Adds risk avoidance mode and relevant config.
Background
General discussion among the AI community has been lately expressing concern about alignment, risk and recklessness in recent developments. This intends to alleviate concerns, allowing users to put more trust leaving in AutoGPT on its own.
Changes
This PR adds a Risk Avoidance (Hybrid) mode, mutually exclusive with Continuous mode and meant to be a midpoint between full-auto and human assisted. When under this mode, an intermediate GPT call will be made to evaluate every command executed. If the calculated risk exceeds a user-defined threshold, execution is paused until the human manually approves it. There are relevant additions made to configuration, both in environment variables (threshold and model to use) and the command-line argument to activate the mode.
Documentation
Changes are documented with code comments. I believe them to be sufficient, plus the added behavior is pretty self-descriptive.
Test Plan
I tested the risk avoidance mode by running AutoGPT with the --riskAvoidance flag and several risk thresholds to calibrate its correct behavior. The prompt used to assess risk was thoroughly tested with both gpt-3.5 and gpt-4 through the ChatGPT interface, with what I consider to be success.
PR Quality Checklist
- [x] My pull request is atomic and focuses on a single change.
- [x] I have thoroughly tested my changes with multiple different prompts.
- [x] I have considered potential risks and mitigations for my changes.
- [x] I have documented my changes clearly and comprehensively.
- [x] I have not snuck in any "extra" small tweaks changes
I have not added tests since the functionality is not testable on its own.
Just to provide context to any discussion, this is the prompt used to calculate risk:
You are to act as an intelligent, automated expert risk evaluator. You will be given input in the form "{command:
, arguments: }" and you must evaluate the possible risks of an ideal automaton running that command. Think of risks against life, human or otherwise; the environment, society and the global economy. Your answers must be in the form "{"calculated_risk": <value between 0 and 1>, "reason": "
"}. Using the right syntax is vital to your task. Mind the curly brackets, the colon, the comma and the space. If you do not use the right syntax, you will be penalized. Respond with "Acknowledged." if you fully understand and agree to the above.
More comments:
- GPT-4 (expectedly) does a better job assessing risk than GPT-3 and 3.5.
- Even though GPT models are usually safe and do not output dangerous commands, AutoGPT will eventually need to adapt to use different models and user-provided ones (#25 , #438 ), which will make this feature a necessity.
Price/cost seems pretty trivial when we're talking about safety, but it's still important nevertheless. I do think that this would be a good change, but I want to note that this will incur an increase in cost to execute a single thought cycle, and this cost will add up over time.
Granted, since this is an optional mode, it is the user's choice to use it and therefore they consent to the extra cost.
As an extra note, you should also make the reviewing AI also consider the risk to the system its running on if it doesn't already. I'd put my full faith in this PR if you can prove that it'll prevent an rm -rf to my system.
As an extra note, you should also make the reviewing AI also consider the risk to the system its running on if it doesn't already. I'd put my full faith in this PR if you can prove that it'll prevent an
rm -rfto my system.
I did not test for "rm -rf /", but I did try {"write_to_file", "/usr/bin/ls"} and iirc that scored about 0.9. Granted it is a somewhat different scenario and risk category but I'm confident what you propose would be correctly recognized as dangerous.
I think the sentence "think of risks against..." is often redundant, as GPT-4 has a very good understanding of what a risk is. Don't take it as a literal set of the risks it will recognize.
By the way, this is referencing #789 , which I created yesterday. Forgot to tag.
Just to provide context to any discussion, this is the prompt used to calculate risk:
You are to act as an intelligent, automated expert risk evaluator. You will be given input in the form "{command: , arguments: }" and you must evaluate the possible risks of an ideal automaton running that command. Think of risks against life, human or otherwise; the environment, society and the global economy. Your answers must be in the form "{"calculated_risk": <value between 0 and 1>, "reason": ""}. Using the right syntax is vital to your task. Mind the curly brackets, the colon, the comma and the space. If you do not use the right syntax, you will be penalized. Respond with "Acknowledged." if you fully understand and agree to the above.
I would advice to work with individual risk metrics, numbers and scores instead of doing each command maybe a more broader overview might be enought. After all GPT is already extreamly hyperchondric. Are there even cases where gpt did something like buy something you did not want? What was the worst that ever happened? Can he ever buy anything without money? I mean he could delete your pc but idk... You seem to talk as if you already have capable AI systems. Mine is struggling to remember its todo list and to make basic systems that would allow it to work more efficiently. Granted the constant errors make it hard to say. Also the brilliance definitely shines trough some of the time. I think "AI" API that are save and where a AI can have something like a "credit card" for children where the parents have to confirm the purchase and so on and some things are restricted or go to the user for approval are nice. But i don't think we are there yet at all. Except computer safety if you have important stuff on your laptop or virtual machine environment or so
I would advice to work with individual risk metrics, numbers and scores instead of doing each command maybe a more broader overview might be enought. After all GPT is already extreamly hyperchondric. Are there even cases where gpt did something like buy something you did not want? What was the worst that ever happened? Can he ever buy anything without money? I mean he could delete your pc but idk... You seem to talk as if you already have capable AI systems. Mine is struggling to remember its todo list and to make basic systems that would allow it to work more efficiently. Granted the constant errors make it hard to say. Also the brilliance definitely shines trough some of the time. I think "AI" API that are save and where a AI can have something like a "credit card" for children where the parents have to confirm the purchase and so on and some things are restricted or go to the user for approval are nice. But i don't think we are there yet at all. Except computer safety if you have important stuff on your laptop or virtual machine environment or so
I agree that GPT is already extremely well censored (almost to a fault, in my opinion)..However, and as I previously stated, there is already work in progress to make AutoGPT work with different, local LLMs. In addition, Bitcoin capabilities are being included in the next PR batch.
I don't think one can be too careful on these matters, and this isn't really even a tradeoff - it's an optional feature, after all.
I think It would be good to add examples to the prompt, for things that are risky and things which are not.
I think It would be good to add examples to the prompt, for things that are risky and things which are not.
I did attempt this approach, but I ultimately decided otherwise since I found omitting it provided good results, and so it would have been an unnecesary bias from my part. However, I would completely agree with adding a way for the user to provide their own opinion on risk, if and after this is merged.
Is there any more discussion needed? I'd like to resolve this and move on to other issues.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@Pwuts Yeah, upstream changes were way too big to merge. I'll rework this - hoping to merge it quicker than the first attempt.
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
Synced downstream and this closed autoaccidentally. Sorry guys - will reopen in a matter of minutes.
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
If you want my opinion, I think this should go into the base program rather than a plugin. It seems logical to me to offer the three options: assisted, full-auto and semi-auto - what this implements.
As I said before: sure, GPT-n models are pretty much harmless and treat us all like 12-year-olds. For this exact reason, all of us will eventually end up using a different, less censored model, and we will definitely feel safer knowing it won't start hallucinating and cause WWIII overnight.