Inconsistent character encoding from windows output

Open jtpio opened this issue 3 months ago • 1 comments

Looking at this issue in the JupyterLab repo, we wanted to migrate it here since it's not related to the JupyterLab frontend: https://github.com/jupyterlab/jupyterlab/issues/1239

Original report:

trying "!dir" in Jupyterlab Notebook, hoping for the best as there may be some "utf-8" standardization thanks to PEP 528 / 529, I see no effect: still badly decoded characters.

I have difficulties to guess from bugs.python.org if something can be done in Jupyterlab (or qtconsole) case: https://bugs.python.org/issue28675

Maybe I don't take the right example to see the PEP 528 / 529 effects.

pep_528-529_where_are_you_hidden

Oct 14 '25 15:10 jtpio

This shouldn't be JupyterLab's problem. The protocol does specify that all text is UTF-8. So it is between IPython (or ipykernel, but more likely IPython), where the bytes->str decoding happens exactly once, and there's an incorrect guess (maybe).

I think the encoding is defined here.

If someone can write a simple bit of Python code that does the right thing just with Popen('dir').communicate(), I think we can (maybe) fix it.

Oct 14 '25 20:10 minrk