OpenIndiana
[WIP] Fix UTF-8 handling
# Ran 1132 tests in 18490.569s - skipped 1 tests.
Skipped tests:
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.
======================================================================
BASELINE MATCH
======================================================================
@citrus-it What do you think about utf-8 test? The only issue I see is that now you should run test suite with UTF-8 locale
@citrus-it It's not enough. Got this exception with patched pkg:
$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) \
Traceback (most recent call last):
File "/usr/bin/pkg", line 5968, in handle_errors
__ret = func(*args, **kwargs)
File "/usr/bin/pkg", line 5954, in main_func
pargs=pargs, **opts)
File "/usr/bin/pkg", line 2276, in install
display_plan_cb=display_plan_cb, logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
logger=logger, **kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
display_plan_cb=display_plan_cb)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
for pd in api_plan_func(**kwargs):
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
log_op_end_all=True)
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
self._img.make_install_plan(**kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
reject_list=reject_list)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
self.__call_imageplan_evaluate(ip)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
ip.evaluate()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3879, in evaluate
self.__evaluate_pkg_plans()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4029, in __evaluate_pkg_plans
ccancel=self.__check_cancel, progtrack=self.__progtrack)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 536, in wrapper
return f(instance, *fargs, **f_kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1588, in prefetch_manifests
mfstlist, excludes)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1681, in _prefetch_manifests_list
dl_path)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1780, in _verify_manifest
mcontent = mf.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 36861: ordinal not in range(128)
It's still not complete, now it dies without exception...
$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) |
pkg: 'ascii' codec can't encode character '\u0151' in position 36861: ordinal not in range(128)
The same issue - dies on writing to /var/pkg/cache/publisher/userland/pkg/crypto%2Fca-certificates/1.0%2C5.11-2019.0.0.7%3A20200310T174048Z/manifest.file.uosjwiqq
Now more issues:
Creating Plan (Checking for conflicting actions): -
Traceback (most recent call last):
File "/usr/bin/pkg", line 5968, in handle_errors
__ret = func(*args, **kwargs)
File "/usr/bin/pkg", line 5954, in main_func
pargs=pargs, **opts)
File "/usr/bin/pkg", line 2276, in install
display_plan_cb=display_plan_cb, logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
logger=logger, **kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
display_plan_cb=display_plan_cb)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
for pd in api_plan_func(**kwargs):
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
log_op_end_all=True)
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
self._img.make_install_plan(**kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
reject_list=reject_list)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
self.__call_imageplan_evaluate(ip)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
ip.evaluate()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3880, in evaluate
self.__merge_actions()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4747, in __merge_actions
self.__find_all_conflicts()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3371, in __find_all_conflicts
offset_dict = self.image._load_actdict(self.__progtrack)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 3881, in _load_actdict
for line in of:
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5250: ordinal not in range(128)
Now we can get another error:
Removing old actions 1/1
Action install failed for 'etc/certs/CA/NetLock_Arany_(Class_Gold)_F\u0151tan\xfas\xedtv\xe1ny.pem' (pkg://userland/crypto/ca-certificates):
UnicodeEncodeError: 'ascii' codec can't encode character '\u0151' in position 42: ordinal not in range(128)
@pyhalov With #93 could we merge this or are we still stuck on one case where it still calls open without encoding?
Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?
Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?
No we needed to remove C locale and switch to C.UTF-8 one should not call IPS with just the C locale. We could make a patch to the python source to fix the if case that checks if the string contains .UTF-8 in locale and simply tell it to assume UTF-8 aswell in those cases.