"str is not valid UTF-8: surrogates not allowed" after "y -10" command

[coolk8]

Duplicates

• [X] I have searched the existing issues

Steps to reproduce 🕹

1. Run Auto-GPT as Docker image via VS Code.
2. Provide goals for Auto-GPT.
3. Write "y" command few times.
4. Write "y -10" command few times.
5. Suddenly Auto-GPT fails after another "y -10" command
6. Auto-GPT terminated

Current behavior 😯

NEXT ACTION:  COMMAND = start_agent ARGUMENTS = {'name': 'DiplomacyChallengeSolver', 'task': 'placeholder'}
Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for Peacemaker-AI...
Input:y -10

**Traceback (most recent call last):
  File "/app/main.py", line 406, in <module>
    memory.add(memory_to_add)
  File "/app/memory/local.py", line 64, in add
    out = orjson.dumps(
          ^^^^^^^^^^^^^
TypeError: str is not valid UTF-8: surrogates not allowed**

Expected behavior 🤔

1. No exception after "y -10" command
2. Workaround for this type of exception to prevent Auto-GPT from termination

Your prompt 📝

# Paste your prompt here

[Pwuts]

Is this still an issue with the latest version?

[NeKum]

I have the same problem

[ZaynJarvis]

same here, after typing y, also in docker

Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ...
Input: y
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/appuser/autogpt/__main__.py", line 5, in <module>
    autogpt.cli.main()
  File "/home/appuser/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/appuser/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/appuser/.local/lib/python3.10/site-packages/click/core.py", line 1635, in invoke
    rv = super().invoke(ctx)
  File "/home/appuser/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/appuser/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/appuser/.local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/appuser/autogpt/cli.py", line 151, in main
    agent.start_interaction_loop()
  File "/home/appuser/autogpt/agent/agent.py", line 184, in start_interaction_loop
    self.memory.add(memory_to_add)
  File "/home/appuser/autogpt/memory/local.py", line 89, in add
    out = orjson.dumps(self.data, option=SAVE_OPTIONS)
TypeError: str is not valid UTF-8: surrogates not allowed

[Pwuts]

Can one of you add a pprint to dump the content of self.data on line 89 here, reproduce the issue, and post the output? https://github.com/Significant-Gravitas/Auto-GPT/blob/14d3ecaae75a80f03cd118b5767e1c6affb4e3cc/autogpt/memory/local.py#L88-L89

[ZaynJarvis]

Can one of you add a pprint to dump the content of self.data on line 89 here, reproduce the issue, and post the output?

https://github.com/Significant-Gravitas/Auto-GPT/blob/14d3ecaae75a80f03cd118b5767e1c6affb4e3cc/autogpt/memory/local.py#L88-L89

trying to reproduce error, I didn't find a pattern that will 100% trigger the issue

[ido777]

I have a fix for this (+ test), will create PR

[Nikoxes70]

On a macOS my solution was to grant a Full Disk Access from the terminal.app Go to "System Settings" -> "Privacy & Security" -> "Full Disk Access", press the plus icon and add the Terminal.app to the list.

[Pwuts]

Closing since this issue thread has been idle for a month; I assume it has been fixed.