File decode fails with special characters

[unmotivatedgene]

Characters like 🎙️ and 🔈cause the file loading to fail. I suspect this is because it expects a pure UTF-8 file.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\USERNAME\miniconda3\envs\aider\Scripts\aider.exe\__main__.py", line 7, in <module>
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\site-packages\aider\main.py", line 293, in main
    coder.run()
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\site-packages\aider\coders\base_coder.py", line 328, in run
    new_user_message = self.run_loop()
                       ^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\site-packages\aider\coders\base_coder.py", line 369, in run_loop
    inp = self.io.get_input(
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\site-packages\aider\io.py", line 152, in get_input
    completer_instance = AutoCompleter(root, rel_fnames, addable_rel_fnames, commands)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\site-packages\aider\io.py", line 43, in __init__
    content = f.read()
              ^^^^^^^^
  File "C:\Users\USERNAME\miniconda3\envs\aider\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 4455: character maps to <undefined>

[binaerverkehr]

I get the same error, but only when using Bash. With cmd (windows) it works perfectly.

[paul-gauthier]

Thanks for trying aider and for reporting this issue. This looks like a text encoding issue and might be related to running on Windows. I will work on a proper solution and get back to you.

[daaddd]

@paul-gauthier it's happening on mac too, both terminal and iterm2 zsh

> add sugar option just like we do with calories, protein, etc `admin/incCommon.
php` `snacks_dml.php` `snacks_view.php` `updateDB.php` `admin/incFunctions.php`
`hooks/get_filtered_data.php` `hooks/snacks-tv.js` `hooks/snacks.php` `templates
/snacks_templateTV_new.html`

hooks/get_filtered_data.php
updateDB.php
admin/incFunctions.php
hooks/snacks.php
hooks/snacks-tv.js
snacks_dml.php
snacks_view.php
Add these files to the chat? y
Traceback (most recent call last):
  File "/Users/david/opt/anaconda3/envs/aider/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/main.py", line 293, in main
    coder.run()
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 328, in run
    new_user_message = self.run_loop()
                       ^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 396, in run_loop
    return self.send_new_user_message(inp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 404, in send_new_user_message
    self.choose_fence()
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 260, in choose_fence
    all_content += Path(fname).read_text() + "\n"
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/pathlib.py", line 1059, in read_text
    return f.read()
           ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 31296: invalid start byte
(aider) david@Davids-MacBook-Air pronacks_built %

[daaddd]

and also

(aider) david@Davids-MacBook-Air pronacks_built % aider            
Model: gpt-4
Git repo: .git
Repo-map: universal-ctags using 1024 tokens
Use /help to see in-chat commands.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> /add admin/incCommon.php snacks_dml.php snacks_view.php admin/incFunctions.php hooks/get_filtered_data.php hooks/snack 
s-tv.js hooks/snacks.php templates/snacks_templateTV_new.html                                                                        

Added admin/incCommon.php to the chat
Added snacks_dml.php to the chat
Added snacks_view.php to the chat
Added admin/incFunctions.php to the chat
Added hooks/get_filtered_data.php to the chat
Added hooks/snacks-tv.js to the chat
Added hooks/snacks.php to the chat
hooks/snacks-tv.js is already in the chat
hooks/snacks.php is already in the chat
Added templates/snacks_templateTV_new.html to the chat
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
  File "/Users/david/opt/anaconda3/envs/aider/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/main.py", line 293, in main
    coder.run()
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 328, in run
    new_user_message = self.run_loop()
                       ^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/coders/base_coder.py", line 369, in run_loop
    inp = self.io.get_input(
          ^^^^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/io.py", line 152, in get_input
    completer_instance = AutoCompleter(root, rel_fnames, addable_rel_fnames, commands)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/david/opt/anaconda3/envs/aider/lib/python3.11/site-packages/aider/io.py", line 43, in __init__
    content = f.read()
              ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 31296: invalid start byte

[paul-gauthier]

I just merged a fix for this issue. You'll have to install the latest from github to pick it up. I'll release it to pypi soon.

Please let me know if you continue to see any unicode errors.

[paul-gauthier]

The fix is now on pypi in v0.8.0 and will be installed if you do pip install --upgrade aider-chat.

[paul-gauthier]

I'm going to close this issue, but feel free to re-open or file a new issue if you want to follow up.