added lib ftfy (fixes text for you), to solve surrogates errors

[ido777]

Background

Fixes #778, surrogates errors

Changes

1. added lib to requirments.txt
2. changed the insertion of the text to the list to be fixed before adding it to the list
3. added test of with problematic test that creates an encoding error but pass the addition to the cache with no error

Documentation

Test Plan

Added dedicated test

PR Quality Checklist

• [x] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [x] I have not snuck in any "extra" small tweaks changes

[ntindle]

Update from master

[ido777]

rebased on master

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[codecov[bot]]

Codecov Report

Patch coverage: 100.00% and project coverage change: -8.37 :warning:

Comparison is base (6fc6ea6) 49.65% compared to head (3a5e85a) 41.29%.

:exclamation: Current head 3a5e85a differs from pull request most recent head 32d0c23. Consider uploading reports for the commit 32d0c23 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2983      +/-   ##
==========================================
- Coverage   49.65%   41.29%   -8.37%     
==========================================
  Files          64       64              
  Lines        2924     2901      -23     
  Branches      492      488       -4     
==========================================
- Hits         1452     1198     -254     
- Misses       1353     1647     +294     
+ Partials      119       56      -63     

Impacted Files	Coverage Δ
autogpt/memory/local.py	96.00% <100.00%> (+0.08%)	:arrow_up:

... and 15 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)			May 2, 2023 5:30am