AI tries to access non-existing files

[ph-ausseil]

Duplicates

• [x] I have searched the existing issues

Steps to reproduce 🕹

I think the use case case is pretty interesting a trigger a log of bug basic functions such as :

• Search file
• Read File
• Write File
• Evaluate

I downloaded python.org plain text doc , and unzipped it in auto_gpt_workspace/python_doc.

https://docs.python.org/3/archives/python-3.11.3-docs-text.zip

I create my AI using default config :

AI Name: PythonGPT
PythonGPT is: produce hight quality and well detailed documentation and tutorial for Python Developpers

Goal 1 : You will make a detailled summary of each file located in "./python_doc/" as if you were an engineer willing to teach another engineer on a new concempt. The summary must be detailed and contains all subtilities required. Then each summary must be saved in a file located in the folder "./python_sumary". Goal 2 : Once the first goal is achieved and you have saved the files. You will identify an instruction that could lead to such content/output. Then each instruction must be saved in a file located in the folder "./python_instruction".

I get a lot of bugs :)

Input: Let's do 5 files to begin, if a file is over 4000 characters including spaces , split the file in chunks of 4000 characters or under.

Current behavior 😯

Attempting to fix JSON by finding outermost brackets 
Error: Invalid JSON, setting it to empty JSON now.
  
 THOUGHTS:  None
REASONING:  None
CRITICISM:  None
SPEAK:  None
Attempting to fix JSON by finding outermost brackets 
Error: Invalid JSON, setting it to empty JSON now.
  
NEXT ACTION:  COMMAND = Error: ARGUMENTS = 'dict' object has no attribute 'replace'
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:

Also encountered several bug as this one during my tests :

Apparently json was fixed. 
NEXT ACTION:  COMMAND = read_file ARGUMENTS = {'file': 'filename'}
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:y
-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
SYSTEM:  Command read_file returned: Error: [Errno 2] No such file or directory: '/Users/admin/Dropbox/Mac/Documents/GitHub/Auto-GPT/auto_gpt_workspace/filename'
  Warning: Failed to parse AI output, attempting to fix.

Expected behavior 🤔

File should be found in the directory.

Your prompt 📝

Welcome to Auto-GPT!  Enter the name of your AI and its role below. Entering nothing will load defaults.
Name your AI:  For example, 'Entrepreneur-GPT'
AI Name: newPythonGPT
newPythonGPT here!  I am at your service.
Describe your AI's role:  For example, 'an AI designed to autonomously develop and run businesses with the sole goal of increasing your net worth.'
newPythonGPT is: Produce hight quality and well detailed documentation and tutorial for Python Developpers
Enter up to 5 goals for your AI:  For example: Increase net worth, Grow Twitter Account, Develop and manage multiple businesses autonomously'
Enter nothing to load defaults, enter nothing when finished.
Goal 1: You will make a detailled summary of each file located in "./python_doc/" as if you were an engineer willing to teach another engineer on a new concempt. The summary must be detailed and contains all subtilities required. Then each summary must be saved in a file located in the folder "./python_sumary". 
Goal 2: 
Using memory of type:  LocalCache
Using Browser:  chrome
 THOUGHTS:  Based on your goal to create detailed summaries of each file in the python_doc directory, you can start by listing the contents of the directory using the 'search_files' command. This will allow you to see all the files in the directory and their file paths. You can use this information to loop through each file, read its contents, and write a summary to a new file in the python_summary directory. You can use the 'read_file' command to read the contents of a file, the 'write_to_file' command to write the summary to a new file, and the 'delete_file' command to delete any files in the python_summary directory that you determine need to be replaced with updated summaries.
REASONING:  Listing the contents of the directory and looping through each file programmatically will save time compared to manually opening and closing each file. Creating new files in the python_summary directory and deleting old ones will ensure the summaries are always up to date.
PLAN: 
-  Use the 'search_files' command to list the contents of the python_doc directory
-  Loop through each file listed and use the 'read_file' command to read each file's contents
-  Use the contents of each file to write a summary using the 'write_to_file' command in a new file in the python_summary directory
-  Use the 'delete_file' command to delete files in the python_summary directory that need to be replaced
CRITICISM:  Make sure to write high-quality summaries that will be helpful to other developers. Consider ways to be concise but comprehensive in your summaries. Also, consider using a more sophisticated summarization algorithm or tool to improve the quality of your summaries.
Attempting to fix JSON by finding outermost brackets 
Apparently json was fixed. 
NEXT ACTION:  COMMAND = search_files ARGUMENTS = {'directory': './python_doc/'}
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:Let's do 5 files to begin, if a file is over 4000 characters including spaces , split the file in two.
SYSTEM:  Human feedback: Let's do 5 files to begin, if a file is over 4000 characters including spaces , split the file in two.
  Warning: Failed to parse AI output, attempting to fix.
 If you see this warning frequently, it's likely that your prompt is confusing the AI. Try changing it up slightly.
Failed to fix AI output, telling the AI.  
Error: Invalid JSON
  Understood. Here is a modified plan to summarize the python_doc files:

1. Use the 'search_files' command to list the contents of the python_doc directory.
2. Loop through each of the first five files listed.
3. Use the 'read_file' command to read each file's contents.
4. If a file is over 4000 characters including spaces, split it in two by using the 'write_to_file' command to write the first 4000 characters to a new file in the python_summary directory and then using the 'append_to_file' command to add the remaining characters to the same file.
5. Otherwise, write a summary of each file's contents using the 'write_to_file' command in a new file in the python_summary directory.
6. Be sure to review each summary for quality before moving on to the next file.

Please note that the plan above only summarizes the first five files in the python_doc directory. If you would like to summarize more files, simply modify step 2 to loop through a different number of files.
Attempting to fix JSON by finding outermost brackets 
Error: Invalid JSON, setting it to empty JSON now.
  
 THOUGHTS:  None
REASONING:  None
CRITICISM:  None
SPEAK:  None
Attempting to fix JSON by finding outermost brackets 
Error: Invalid JSON, setting it to empty JSON now.
  
NEXT ACTION:  COMMAND = Error: ARGUMENTS = 'dict' object has no attribute 'replace'
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input:

[villesau]

Might be the same problem as this one: https://github.com/Significant-Gravitas/Auto-GPT/issues/1889

[Pwuts]

Are you by any chance using GPT-3.5? These are issues caused by the language model not generating commands properly. Auto-GPT can't fix that directly.

[ph-ausseil]

villesau

Might be the same problem as this one: #1889

I really don't think so as Auto-GPT is firstly unable to read find file in the proper directory in my case and goes in error, I guess mine is more linked to control on the data returned by search_file or on the repository where search_file is performed.

But's that's to begin with my use case scenario trigger a bunch of issues (JSON length, number of token...) so check should be done on the data sent to the LLM as much as on the data received.

[ph-ausseil]

Are you by any chance using GPT-3.5? These are issues caused by the language model not generating commands properly. Auto-GPT can't fix that directly.

Do you suggest I set my fast FAST_LLM_MODEL as gpt-4 ? I would understand you ! Still I think that adequate reaction need to be put in such case, eventually offer the user tu input a path, and loop if the dir doesn't exist ? search_files is a pretty basic/essential function so this king of safeguard seems important to me.

[ph-ausseil]

I would also outline that the LLM used for each query should appear in error.log , this may help for debugging.

[Pwuts]

Are you by any chance using GPT-3.5? These are issues caused by the language model not generating commands properly. Auto-GPT can't fix that directly.

Do you suggest I set my fast FAST_LLM_MODEL as gpt-4?

What LLM are you currently using?

[ph-ausseil]

Are you by any chance using GPT-3.5? These are issues caused by the language model not generating commands properly. Auto-GPT can't fix that directly.

Do you suggest I set my fast FAST_LLM_MODEL as gpt-4?

What LLM are you currently using?

I've not been invited for the gpt-4 API, so I am on 3.5, but still doesn't make the issue less concerning, proper checking and safeguards.

[Pwuts]

There is checking in Auto-GPT, but the GPT-3.5 model has its limitations and I think those are the cause of your problems.

[ph-ausseil]

There is checking in Auto-GPT, but the GPT-3.5 model has its limitations and I think those are the cause of your problems.

Obviously there is no safeguard there, when Auto-GPT look for "/Users/admin/Dropbox/Mac/Documents/GitHub/Auto-GPT/auto_gpt_workspace/filename" , and crash because it doesn't find any document. A safeguard would be to return an anomaly and offer the user to confirm the input (/Users/admin/Dropbox/Mac/Documents/GitHub/Auto-GPT/auto_gpt_workspace/filename) , change the input or force to continue. Such, error can't be put on the model because model are no perfect as dev we are here to put safegard on wrong input . Inputs from Human or LLM 😸 .

[AndroidDev77]

I get the same thing, it's not able to read or write files

[Pwuts]

@ph-ausseil fair enough, it could be useful to have some graceful error handling for cases like that. Even if it's just reformulating the error to let the AI know exactly what it is doing wrong.

[ph-ausseil]

@ph-ausseil fair enough, it could be useful to have some graceful error handling for cases like that. Even if it's just reformulating the error to let the AI know exactly what it is doing wrong.

An interesting paramater would be SELF_RECTIFICATION which if no let human interveen to fix an issue or Cue the AI to fix itself.

Also it may be interesting that error.log log which model is being queried,

[Pwuts]

Something like that seems to be implemented here. Maybe it could be expanded upon.

[ph-ausseil]

I see, that’s a choice with regards to implementation.

Autocorrection while desirable may not always be feasible. There could be a choice to be made between full autonomy & semi autonomy for error handling.

Feasability put aside, I guess on the principle it’s interesting to handle both deviations & incorrect/unexpected inputs in this situation and in other.

[ph-ausseil]

Something like that seems to be implemented here. Maybe it could be expanded upon.

Also a though, but it could be interesting for #15

[ph-ausseil]

Something like that seems to be implemented here. Maybe it could be expanded upon.

I have forked the master & tried to implement automated fix, however I'm not good at prompt engineering so I have not implemented this part. For the moment, it can offer manual corrections. Thoughs ?

https://github.com/ph-ausseil/Auto-GPT/commit/07f4f67ebd8760958c9184322552e54cb1e7e358

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.