Should init() be wrapping stdout on Linux?

[DanielFEvans]

When using colorama in a non-interactive terminal (e.g. stdout being piped to file) on Linux, colorama.init() is hooking into sys.stdout, resulting in occasional strange results.

Looking at the documentation, there are a few lines that state:

"Colorama makes this work on Windows, too, by wrapping stdout [...] On other platforms, Colorama does nothing."

Example colorama_test.py script:

import sys
import colorama

print(sys.stdout)
colorama.init()
print(sys.stdout)

If I run python colorama_test.py in a bash terminal, colorama makes no change to sys.stdout:

<open file '<stdout>', mode'w' at 0x7f9f178ab150>
<open file '<stdout>', mode'w' at 0x7f9f178ab150>

However, if I run python colorama_test.py > out.txt, sys.stdout is wrapped:

<open file '<stdout>', mode'w' at 0x7f75cabe1150>
<colorama.ansitowin32.StreamWrapper object at 0x7f75caaa0f10>

Should this be happening?

For completeness, the "occasional strange results" I see occur when writing to stdout via stdout.write; partial traceback is:

[...]
    sys.stdout.write(struct.pack('i', value))
  File "/home/jbanorthwest.co.uk/danielevans/venvs/farmcat/lib/python2.7/site-packages/colorama/ansitowin32.py", line 41, in write
    self.__convertor.write(text)
  File "/home/jbanorthwest.co.uk/danielevans/venvs/farmcat/lib/python2.7/site-packages/colorama/ansitowin32.py", line 162, in write
    self.write_and_convert(text)
  File "/home/jbanorthwest.co.uk/danielevans/venvs/farmcat/lib/python2.7/site-packages/colorama/ansitowin32.py", line 184, in write_and_convert
    text = self.convert_osc(text)
  File "/home/jbanorthwest.co.uk/danielevans/venvs/farmcat/lib/python2.7/site-packages/colorama/ansitowin32.py", line 256, in convert_osc
    winterm.set_title(params[1])
AttributeError: 'NoneType' object has no attribute 'set_title'

[DanielFEvans]

Aha, I at least see why my "strange result" was happening - I'm writing encoded binary to file, and must have passed the magic combination of bytes that colorama interprets as a request to change the window title.

Also, I admit it's odd to be importing colorama in this situation; it's being eagerly init'ed by a dependency-of-a-dependency.

[gwtwod]

I'm curious about this behavior on Linux as well. I was making a script with an option to force colored output when writing to a non-TTY, and happened to import Structlog which in turn calls colorama.init()

That caused all my ANSI-codes to get stripped no matter what and wasted a few hours before I discovered my sys.stdout was being overwritten when piping/redirecting output.

Is this really intended?

[wiggin15]

@gwtwod Wrapping stdout on Linux when redirection is enabled is indeed intended. On Windows we strip ANSI codes from redirected output because ANSI codes won't work on that output, so we also do that for Linux - and many times, it is a desired behavior. If it is not, it's possible to pass strip=False, or wrap=False, to colorama.init.

@DanielFEvans the problem with winterm.set_title looks like a bug. On Unix systems we shouldn't be trying to call the Windows API to set the title. The redirected output will cause the stream to be wrapped, but the flag that determines whether to call the Win32 API is convert and we don't check for that flag when we find the ANSI sequence for "set title".

[wiggin15]

I pushed a fix to my personal forked repository - https://github.com/wiggin15/colorama/commit/4832ef940d486b9d6ae74795b2df2b59a78d6075 I don't have time to properly test this though so I'm holding off merging this.

[ossdev07]

@wiggin15 I am working on adding aarch64 support in colorama, following two test cases are failing in my local environment:

======================================================================
FAIL: testInitDoesntWrapOnEmulatedWindows (colorama.tests.initialise_test.InitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.5/lib/python3.7/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)
  File "/home/travis/build/ossdev07/colorama/colorama/tests/initialise_test.py", line 50, in testInitDoesntWrapOnEmulatedWindows
    self.assertNotWrapped()
  File "/home/travis/build/ossdev07/colorama/colorama/tests/initialise_test.py", line 35, in assertNotWrapped
    self.assertIs(sys.stdout, orig_stdout, 'stdout should not be wrapped')
AssertionError: <colorama.ansitowin32.StreamWrapper object at 0xffffb14dfdd0> is not <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> : stdout should not be wrapped
======================================================================
FAIL: testInitDoesntWrapOnNonWindows (colorama.tests.initialise_test.InitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/ossdev07/colorama/colorama/tests/initialise_test.py", line 55, in testInitDoesntWrapOnNonWindows
    self.assertNotWrapped()
  File "/home/travis/build/ossdev07/colorama/colorama/tests/initialise_test.py", line 35, in assertNotWrapped
    self.assertIs(sys.stdout, orig_stdout, 'stdout should not be wrapped')
AssertionError: <colorama.ansitowin32.StreamWrapper object at 0xffffb14dfed0> is not <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> : stdout should not be wrapped

Please shed some light on this, what could possibaly be the reason for the test failures.

[wiggin15]

@ossdev07 I think you should open a separate ticket for this problem. Also, please share the command you are running (is it make test?)

[barabanus]

Hello, I think that when we call colorama.init() we should specify which output we'd like to "wrap". For example, in my framework I've got a lot of scripts which use stderr for logging, but stdout to generate binary files.

Ok, I can set colorama.init(wrap=False) as I don't need it to work in Windows, but wouldn't it be useful to have a separate keyword to permit stderr wrapping, but not stdout?

UPDATE 1:

actually, colorama.init(wrap=False) or colorama.init(strip=False) is not an option because it doesn't cut color codes when output is redirected.

UPDATE 2:

Currently I init colorama like this:

# init colorama to strip color codes from stderr, but not stdout
stdout = sys.stdout
colorama.init()
sys.stdout = stdout

But please, make it possible to do without dirty hacks. For example, it may look like this:

colorama.init()
colorama.init(wrap="stdout")
colorama.init(wrap="stderr")
colorama.init(wrap=False)

[Delgan]

@barabanus I guess you can manually wrap stdout if you need to.

from colorama import AnsiToWin32

sys.stdout = AnsiToWin32(sys.stdout, convert=True, strip=False, autoreset=False).stream

[barabanus]

@barabanus I guess you can manually wrap stdout if you need to.

Thank you for your advice, but obviously, it's too wordy. Also, it's not like I care at all about windows users to add specific wrapper. So I would prefer something like colorama.init(wrap="stderr"). I think it's quite elegant solution given the current interface.