update_file method to change occurences of text in a file

[bfalans]

Background

Auto-gpt is not able to edit files, the addtion of an update_file method would tackle this issue old PR: #3464

Changes

addition of an update_file method that can change one or all occurences of specified text

Documentation

Inline documentation provided for the function

Test Plan

2 tests written in unittest Test_file_operations.py and verified

PR Quality Checklist

• [x] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [x] I have not snuck in any "extra" small tweaks changes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		Jun 3, 2023 10:34pm

[Boostrix]

I am all in when it comes to formalizing these things, but I have seen it use CLI editors like sed/awk and perl to edit existing files in a rather clever way, at least on *nix. So this may boil down to the OS in use. Note that one recurring "issue" reported frequently on the issue tracker is that Auto-GPT starts interactive editors like vim/nano or joe to edit stuff - so it's definitely trying to do the right thing. Merely failing at NOT using interactive apps, and failing at knowing more about the OS in question, and the tooling/version numbers available.

On a Linux system, if augment the shell_execute description to add a corresponding constraint, you will definitely see much better results.

Then again, like I said, using a dedicated update_file/edit_file command sounds definitely like a good idea, and it would also be the right place to encode preference for CLI tools like sed/awk there.

addition of an update_file method that can change all occurences of specified text

not having looked at the code/patch in question, I find that rather limited if that's indeed what it is doing. Like I said, for people on Linux, Auto-GPT can definitely be told to update/edit files using standard CLI toolks like sed/vi or even perl (non-interactively). Thus, if you need just a search/replace command, my suggestion would be to use a regex wrapper for that - that way it's broader in scope, and the backend could still be using sed on these systems.

[bfalans]

@boostrix, I found that on a windows system, autogpt is not able to use any programs the right way afaik. Yes it is a basic function, but we need to start somewhere and keep updates atomized.

[Boostrix]

For cross-platform file editing functionality similar to sed and awk, you can use Python's built-in string manipulation and regular expression modules: re, along with its file I/O functions.

[bfalans]

@Boostrix would you care to help me adjust it? i am not that known to all the python packages yet and am learning a lot from help provided.

[Boostrix]

assuming that you have access to GPT, you can probably just copy your changes and ask it to adapt those to use the Python "re" module instead - it's a fairly self-contained change since your code is already using the re module.

[bfalans]

It is already using re, so I will try to adjust it. I have access to gpt but only 3.5

[bfalans]

@Boostrix I tried chat gpt, but it comes up with the same code constantly only changing re.finditer to re.sub. I don't think this is what you meant 😅.

[bfalans]

function changed to use re for cross_platform compatibilty Function now changes one or all occurences of specified text

[Boostrix]

function changed to use re for cross_platform compatibilty Function now changes one or all occurences of specified text

my 2c: I still believe having a lower-level function to run a regex on an input/file analogous to sed/awk would be more powerful and more general, higher level functions like "replace_text" could be built on top of this, simply by invoking the regex command. The "workhorse" portion of this code should not be replacing a text in a file, but it should be running a regex - so that other, higher level, command would simply specify the regex/file name to be used when invoking this.

Don't get me wrong, this is great and useful "as is" - but it could be MUCH more useful by using the regex/file approach - also to edit stuff in place.

[bfalans]

@boostrix I will take a look, but could use some help as my programming skills are not up-to-date yet

[Boostrix]

@Boostrix I will take a look, but could use stem help as my programming skiles are not up-to-date yet

I would not expend any time/effort on this, unless others agree with my opinion. But the idea itself is rather simple: rather than run a search/replace, just run the regex.

The search/replace would then be built on top of your regex command, which would then be much simpler/shorter

[bfalans]

@boostrix great new coding experience for me to look into that, but for now I hope someone can review this so it can be implemented or so I know what to change. I can change it when I have new code ready

[bfalans]

Executing file '/app/autogpt/auto_gpt_workspace/retrieve_ticker_data.py' SYSTEM: Command execute_python_file returned: Error: File "/app/autogpt/auto_gpt_workspace/retrieve_ticker_data.py", line 13 print(btc_usd_ticker)import collections.abc ^^^^^^ SyntaxError: invalid syntax THOUGHTS: There is another syntax error in the Python script. I will use the 'update_file' command to fix the error by adding a newline character between the 'print' statement and the 'import' statement. REASONING: I will use the 'update_file' command to fix the error by adding a newline character between the 'print' statement and the 'import' statement. PLAN:

• Use the 'update_file' command to fix the error by adding a newline character between the 'print' statement and the 'import' statement. CRITICISM: I need to ensure that the error is fixed and that the Python script executes successfully. NEXT ACTION: COMMAND = update_file ARGUMENTS = {'filename': '/app/autogpt/auto_gpt_workspace/retrieve_ticker_data.py', 'old_text': 'print(btc_usd_ticker)import collections.abc', 'new_text': 'print(btc_usd_ticker)\nimport collections.abc', 'occurrence_index': 0} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Asking user via keyboard... Input:do not use cbpro but an alternative SYSTEM: Human feedback: do not use cbpro but an alternative THOUGHTS: I have received feedback to use an alternative to the 'cbpro' library. I will use the 'update_file' command to replace the 'cbpro' library with the 'ccxt' library, which is a popular alternative for cryptocurrency trading. REASONING: I will use the 'update_file' command to replace the 'cbpro' library with the 'ccxt' library, which is a popular alternative for cryptocurrency trading. PLAN:
• Use the 'update_file' command to replace the 'cbpro' library with the 'ccxt' library, which is a popular alternative for cryptocurrency trading. CRITICISM: I need to ensure that the 'ccxt' library is installed and that the Python script executes successfully. NEXT ACTION: COMMAND = update_file ARGUMENTS = {'filename': '/app/autogpt/auto_gpt_workspace/retrieve_ticker_data.py', 'old_text': 'import cbpro', 'new_text': "import ccxt\n\n# Connect to the Coinbase API\npublic_client = ccxt.coinbasepro()\n\n# Retrieve the latest ticker data for the cryptocurrency pair(s) you wish to trade\nbtc_usd_ticker = public_client.fetch_ticker('BTC/USD')\n\n# Print the latest ticker data\nprint(btc_usd_ticker)", 'occurrence_index': 0} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Asking user via keyboard... Input:y -=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= SYSTEM: Command update_file returned: File /app/autogpt/auto_gpt_workspace/retrieve_ticker_data.py updated successfully

as an example that is running as we speak

[bfalans]

@Pwuts requested changes for whitespace and print statements applied. added a test to show no problems are created with multi-line input ending in newline character.

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[Boostrix]

@bfalans been playing with this, and it does make a difference. However, the logging output when updating a file could probably highlight differences between two files, i.e. by coloring additions/removals differently or alternatively by showing an actual diff or at least a summary (added/removed xx words/lines). What do you think ?

Inspired by:

• #4045

[bfalans]

@boostrix the logging at this moment only tries to prevent double actions. I think the showing of changes would maybe be more of a plugin using an external program or own code

[Boostrix]

I think the showing of changes would maybe be more of a plugin using an external program or own code

See #4079

[Boostrix]

I can change it when I have new code ready

for future changes to the update_file command, you might be interested in the python "sed" module: https://pypi.org/project/pythonsed/ You would get all of sed functionality in a cross-platform fashion, not just search/replace. And if the command mentiony "pysed" inside its description, the LLM can suggest appropriate expressions too.

[lc0rp]

I can change it when I have new code ready

for future changes to the update_file command, you might be interested in the python "sed" module: pypi.org/project/pythonsed You would get all of sed functionality in a cross-platform fashion, not just search/replace. And if the command mentiony "pysed" inside its description, the LLM can suggest appropriate expressions too.

Nice. I wonder whether this is a good option for a plugin.

[github-actions[bot]]

We committed some changes to this branch, please run

git fetch
git rebase origin/update_file

before pushing more changes to the remote.

[codecov[bot]]

Codecov Report

Patch coverage: 75.00% and project coverage change: +0.04 :tada:

Comparison is base (63b79a8) 69.65% compared to head (b6b3333) 69.69%.

:exclamation: Current head b6b3333 differs from pull request most recent head 5435ca1. Consider uploading reports for the commit 5435ca1 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3643      +/-   ##
==========================================
+ Coverage   69.65%   69.69%   +0.04%     
==========================================
  Files          72       72              
  Lines        3523     3551      +28     
  Branches      562      569       +7     
==========================================
+ Hits         2454     2475      +21     
- Misses        881      885       +4     
- Partials      188      191       +3     

Impacted Files	Coverage Δ
autogpt/commands/file_operations.py	82.32% <75.00%> (-1.21%)	:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[github-actions[bot]]

We committed some changes to this branch, please run

git fetch
git rebase origin/update_file

before pushing more changes to the remote.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size