PackageDev icon indicating copy to clipboard operation
PackageDev copied to clipboard

`packagedev_convert` can't handle unicode characters when passed `ensure_ascii: false`

Open AmjadHD opened this issue 2 months ago • 2 comments

Running:

>>> window.run_command("packagedev_convert", {"target_format": "json", "ext": "tmLanguage.json", "ensure_ascii": False})

on test.YAML-tmLanguage:

---
scopeName: source.test

patterns:
  - match: "★"

results in an incomplete test.tmLanguage.json:

{
    "scopeName": "source.test",
    "patterns": [
        {
            "match": 

with an error:

Traceback (most recent call last):
  File "C:\Program Files\Sublime Text\Lib\python38\sublime_plugin.py", line 1599, in run_
    return self.run(**args)
  File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/file_conversion.py", line 233, in run
  File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/lib/fileconv/dumpers.py", line 177, in dump
  File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/lib/fileconv/dumpers.py", line 269, in write
  File "./python3.8/json/__init__.py", line 180, in dump
  File "./python3.8/encodings/cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\u2605' in position 1: character maps to <undefined>

OS: Windows 10

AmjadHD avatar Dec 12 '25 21:12 AmjadHD

>>> text = view.substr(sublime.Region(0, view.size())); text
'---\nscopeName: source.test\n\npatterns:\n  - match: "★"\n'
>>> obj = yaml.safe_load(text); obj
{'scopeName': 'source.test', 'patterns': [{'match': '★'}]}
>>> json.dumps(obj)
'{"scopeName": "source.test", "patterns": [{"match": "\\u2605"}]}'
>>> json.dumps(obj, ensure_ascii=False)
'{"scopeName": "source.test", "patterns": [{"match": "★"}]}'
>>> import locale; locale.getpreferredencoding()
'cp1252'

AmjadHD avatar Dec 12 '25 21:12 AmjadHD

Adding encoding='utf-8' to open fixes it: https://github.com/SublimeText/PackageDev/blob/004eb12d2dc58ef2078566c748c800e5f67bc9c0/plugins/lib/fileconv/dumpers.py#L268

AmjadHD avatar Dec 12 '25 22:12 AmjadHD