doctest outputs ANSI escapes to non-color TTYs, resulting in visible junk

[snej]

Description

doctest’s check for whether to enable color output is too simplistic: it just tests isatty(STDOUT_FILENO) (e.g. in the color_to_stream function.) But not all TTYs support ANSI color; one common exception is Xcode’s console pane.

This causes junk to appear in the output for just about anyone developing native Mac or iOS apps unless they disable color on the command line.

Steps to reproduce

Run doctest tests in an Xcode project, with output going to the console pane as usual.

• Expected: Plain output
• Actual: Lots of “[0;36m and the like in the console output

Extra information

On Unix systems you can test the TERM environment variable, but that doesn’t work on Windows. Here’s a reliable function that I’ve used in other projects:

static bool isColor(int fd) {
    if (!isatty(fd))
        return false;

    if (const char *term = getenv("TERM")) {
        if (strstr(term,"ANSI") || strstr(term,"ansi") || strstr(term,"color"))
            return true;
    }

#ifdef _MSC_VER
#ifndef ENABLE_VIRTUAL_TERMINAL_PROCESSING
#define ENABLE_VIRTUAL_TERMINAL_PROCESSING  0x0004
#endif
    if (GetRealOSVersion().dwMajorVersion >= 10) {
        HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);
        DWORD consoleMode;
        if(GetConsoleMode(hConsole, &consoleMode)) {
            SetConsoleMode(hConsole, consoleMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING);
        }
        return true;
    }
#endif

    return false;
}