api-cookbook icon indicating copy to clipboard operation
api-cookbook copied to clipboard

[Bug]: Python on Windows defaults to cp1252 encoding for CSV files

Open alexc-rw opened this issue 10 months ago • 1 comments

Do I have the most recent api-cookbook code?

  • [x] I am using the latest code available on the main branch of the api-cookbook repo.

Is there an existing issue for this?

  • [x] I have searched the existing issues.

Current Behavior

Running as-is results in an encoding exception:

 pipenv run python pivotal_import.py
[Note] This importer adheres to the Shortcut API rate limit of 200 requests per minute.
       It may pause for up to 70 seconds during processing to avoid request throttling.
Traceback (most recent call last):
  File "C:\Users\user\Documents\DEV\shortcut-api-cookbook\pivotal-import\pivotal_import.py", line 664, in <module>
    sys.exit(main(sys.argv))
             ~~~~^^^^^^^^^^
  File "C:\Users\user\Documents\DEV\shortcut-api-cookbook\pivotal-import\pivotal_import.py", line 655, in main
    process_pt_csv_export(ctx, cfg["pt_csv_file"], entity_collector)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Documents\DEV\shortcut-api-cookbook\pivotal-import\pivotal_import.py", line 596, in process_pt_csv_export
    header = [col.lower() for col in next(reader)]
                                     ~~~~^^^^^^^^
  File "C:\Users\user\AppData\Local\Programs\Python\Python313\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5110: character maps to <undefined>

Fix:

Change line 594 of pivotal_import.py to specify utf-8:

with open(pt_csv_file, mode="r", encoding="utf-8") as csvfile:

Expected Behavior

Import running as expected

Minimal Reproducible Example

pipenv run python pivotal_import.py in Windows Powershell with Py 3.13

Environment

  • Operating system: Win11
  • Python/JavaScript version 3.13

Further Information

No response

alexc-rw avatar Jan 23 '25 02:01 alexc-rw