UTF8 files unsupported

[PovilasID]

⚠️ Search for existing issues first ⚠️

• [X] I have searched the existing issues, and there is no existing issue for my problem

Which Operating System are you using?

Linux

Which version of Auto-GPT are you using?

Stable (branch)

GPT-3 or GPT-4?

GPT-4

Steps to reproduce 🕹

Place a file with UTF8 encoding in auto_gpt_workspace folder and ask it to read it

Current behavior 😯

Gets into a loop trying to read the file:

SYSTEM:  Command read_file returned: Error: 'utf-8' codec can't decode byte 0xa0 in position 1341: invalid start byte

Expected behavior 🤔

Well... to read the file

Your prompt 📝

ai_goals:
- Read project-plan-form.htm file 
- Fill project-plan-form.htm for and idea with flying monkeys
ai_name: DocWritter
ai_role: Fill in a doc using a file template

Your Logs 📒

Using memory of type:  LocalCache
Using Browser:  chrome
 THOUGHTS:  I need to fill in the project-plan-form.htm file for an idea with flying monkeys. I should start by reading the file template to understand the required fields.
REASONING:  Before I can fill in the form, I need to understand the required fields. This will help me to avoid errors and ensure that the form is filled correctly.
PLAN: 
-  Read the project-plan-form.htm file
-  Identify the required fields
-  Fill in the form with the required information
CRITICISM:  I need to be careful to fill in all the required fields and avoid making any errors.
NEXT ACTION:  COMMAND = read_file ARGUMENTS = {'file': 'project-plan-form.htm'}
SYSTEM:  Command read_file returned: Error: 'utf-8' codec can't decode byte 0xa0 in position 1341: invalid start byte
              
Aborted!

[k-boikov]

I think the file was written in cp1252 but it tries to read it in utf8.

[sidewaysthought]

The code as-is only functions with UTF-8. If it is in any other format, the code falls over. I've updated the code to detect the file encoding, then use that encoding to read the file.

[Pwuts]

Duplicate of #232