PackageDev
PackageDev copied to clipboard
`packagedev_convert` can't handle unicode characters when passed `ensure_ascii: false`
Running:
>>> window.run_command("packagedev_convert", {"target_format": "json", "ext": "tmLanguage.json", "ensure_ascii": False})
on test.YAML-tmLanguage:
---
scopeName: source.test
patterns:
- match: "★"
results in an incomplete test.tmLanguage.json:
{
"scopeName": "source.test",
"patterns": [
{
"match":
with an error:
Traceback (most recent call last):
File "C:\Program Files\Sublime Text\Lib\python38\sublime_plugin.py", line 1599, in run_
return self.run(**args)
File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/file_conversion.py", line 233, in run
File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/lib/fileconv/dumpers.py", line 177, in dump
File "C:\Users\User\AppData\Roaming\Sublime Text\Installed Packages\PackageDev.sublime-package\plugins/lib/fileconv/dumpers.py", line 269, in write
File "./python3.8/json/__init__.py", line 180, in dump
File "./python3.8/encodings/cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\u2605' in position 1: character maps to <undefined>
OS: Windows 10
>>> text = view.substr(sublime.Region(0, view.size())); text
'---\nscopeName: source.test\n\npatterns:\n - match: "★"\n'
>>> obj = yaml.safe_load(text); obj
{'scopeName': 'source.test', 'patterns': [{'match': '★'}]}
>>> json.dumps(obj)
'{"scopeName": "source.test", "patterns": [{"match": "\\u2605"}]}'
>>> json.dumps(obj, ensure_ascii=False)
'{"scopeName": "source.test", "patterns": [{"match": "★"}]}'
>>> import locale; locale.getpreferredencoding()
'cp1252'
Adding encoding='utf-8' to open fixes it:
https://github.com/SublimeText/PackageDev/blob/004eb12d2dc58ef2078566c748c800e5f67bc9c0/plugins/lib/fileconv/dumpers.py#L268