On Windows under Python 3, Colorama's ANSI color replacement doesn't work in cmd.exe

[acastaner]

Problem

When running beets in a Windows 10 command-line terminal (cmd.exe) the output is wrong. It seems like there's a character encoding error. This is what it looks like (look at the last line where beets prompts for Skipping, Replace, etc) : https://imgur.com/a/b9K9x

E:\Music\_trier>beet -vv
user configuration: C:\Users\Arnaud\AppData\Roaming\beets\config.yaml
data directory: C:\Users\Arnaud\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
inline: adding item field initial
inline: adding item field disc_and_track
library database: C:\Users\Arnaud\AppData\Roaming\beets\library.db
library directory: E:\Music
Sending event: library_opened
Usage:
  beet COMMAND [ARGS...]
  beet help COMMAND

Options:
  --format-item=FORMAT_ITEM
                        print with custom format
  --format-album=FORMAT_ALBUM
                        print with custom format
  -l LIBRARY, --library=LIBRARY
                        library database file to use
  -d DIRECTORY, --directory=DIRECTORY
                        destination music directory
  -v, --verbose         log more details (use twice for even more)
  -c CONFIG, --config=CONFIG
                        path to configuration file
  -h, --help            show this help message and exit

Commands:
  acousticbrainz    fetch metadata from AcousticBrainz
  clearart          remove images from file metadata
  config            show or edit the user configuration
  embedart          embed image files into file metadata
  extractart        extract an image from file metadata
  fetchart          download album art
  fields            show fields available for queries and format strings
  fingerprint       generate fingerprints for items without them
  ftintitle         move featured artists to the title field
  help (?)          give detailed help on a specific sub-command
  import (imp, im)  import new music
  lastgenre         fetch genres
  list (ls)         query the library
  modify (mod)      change metadata fields
  move (mv)         move or copy items
  remove (rm)       remove matching items from the library
  replaygain        analyze for ReplayGain
  stats             show statistics about the library or a query
  submit            submit Acoustid fingerprints
  update (upd, up)  update the library
  version           output version information
  write             write tag information to files
Sending event: cli_exit

Setup

• OS: Windows 10 Professional
• Python version: 3.6.1
• beets version: 1.4.5
• Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

E:\Music\_trier>beet config
directory: E:\Music

plugins: inline acousticbrainz chroma embedart fetchart lastgenre replaygain ftintitle
threaded: yes
autotag: yes
art_filename: cover
asciify_paths: yes

clutter:
- .jpg
- .jpeg
- .png
- .sfv
- .nfo
- .m3u
- .m3u8
- Thumbs.DB
- .DS_Store
original_date: yes

format_item: $artist - $album - $title ($genre)

match:
    strong_rec_thresh: 0.04
    preferred:
        countries: [US, GB|UK, FR]
        media: [CD, Digital Media|File]
original_year: yes
item_fields:
    initial: albumartist[0].upper() + u'.'
    disc_and_track: u'%02i-%02i' % (disc, track) if disctotal > 1 else u'%02i' % (track)

paths:
    albumtype:soundtrack: ost/$album/$disc_and_track. $title
    savedir:arthur: _arthur/$album/$disc_and_track. $title
    savedir:radionova: nova/$album/$disc_and_track. $artist - $title
    savedir:world: world/$albumartist/$year - $album/$disc_and_track. $title
    savedir:electro: electro/$albumartist/$year - $album/$disc_and_track. $title
    savedir:classical: classical/$artist/$album/$disc_and_track. $title
    savedir:alt_rock: alt_rock/$albumartist/$year - $album/$disc_and_track. $title
    savedir:rock: rock/$albumartist/$year - $album/$disc_and_track. $title
    savedir:jazz: jazz/$albumartist/$year - $album/$disc_and_track. $title
    savedir:blues: blues/$albumartist/$year - $album/$disc_and_track. $title
    savedir:metal: metal/$albumartist/$year - $album/$disc_and_track. $title
    savedir:rap: rap/$albumartist/$year - $album/$disc_and_track. $title
    savedir:pop: pop/$albumartist/$year - $album/$disc_and_track. $title
    savedir:techno: techno/$albumartist/$year - $album/$disc_and_track. $title
    default: _trier/$albumartist/$year - $album/$disc_and_track. $title
    comp: $genre/$album/$track. $title

ui:
    color: yes
    colors:
        text_success: green
        text_warning: yellow
        text_error: red
        text_highlight: red
        text_highlight_minor: lightgray
        action_default: turquoise
        action: blue

import:
    write: yes
    copy: yes
    move: yes
    resume: ask
ftintitle:
    auto: yes
    drop: no
    format: feat. {0}
acousticbrainz:
    auto: yes
    force: no
chroma:
    auto: no
acoustid:
    apikey: REDACTED
embedart:
    auto: yes
    ifempty: yes
    maxwidth: 0
    compare_threshold: 0
    remove_art_file: no
fetchart:
    auto: yes
    cover_names: cover front art album folder
    sources: filesystem coverart itunes amazon albumart fanarttv
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no
lastgenre:
    auto: yes
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    separator: ', '
    prefer_specific: no
lastfm:
    user: redacted
replaygain:
    backend: bs1770gain
    auto: yes
    overwrite: no
    targetlevel: 89
    r128: [Opus]
    chunk_at: 5000
    method: replaygain
copyartifacts:
    extensions: .cue .log
    print_ignored: yes
pathfields: {}
album_fields: {}

[sampsyo]

Hi! Those are console color control sequences. They're supposed to be automatically adapted into control commands for the Windows console.

Can you try running this command to check which platform beets believes it's running on?

python -c 'import sys; print(sys.platform)'

And this one to check whether Colorama, which provides colors on Windows, was successfully installed?

python -c 'import colorama; colorama.init()'

If Colorama is missing, you can install it with pip install colorama. But it should have been automatically installed when you installed beets, so that remains a mystery.

In the mean time, you can make the output more readable by setting color: no.

[acastaner]

Apparently cmd.exe doesn't like these commands :/

C:\Users\Arnaud>python -c 'import sys; print(sys.platform)'
  File "<string>", line 1
    'import
          ^
SyntaxError: EOL while scanning string literal

Disabling the colors works as a workaround though!

[sampsyo]

Sorry; I'm very far from being an expert in Windows command syntax. We just need a way to escape the spaces in that argument—maybe try double quotes instead of single quotes?

[acastaner]

No worries :)

The output of the first command is win32. The outpout of the second command is blank but colorama is apparently already installed:

C:\Users\Arnaud>pip install colorama Requirement already satisfied: colorama in c:\users\arnaud\appdata\local\programs\python\python36-32\lib\site-packages

Note that within PowerShell there's no such error (it interprets the error correctly) ; but I tend to use cmd because PS has issues parsing the directory parameter (that's for another issue ;) )

[sampsyo]

That's strange. All this indicates that everything should be working.

Here's the relevant bit of the code: https://github.com/beetbox/beets/blob/7e8056b0e8652a6235d8e2e054c0c41eae71bff0/beets/ui/init.py#L47-L54

If you're interested in doing some more hardcore digging, the next step would be to sprinkle in some print() calls around there to check exactly what's going on. Something like this:

if sys.platform == 'win32':
    try:
        print('importing')
        import colorama
        print('imported')
    except ImportError:
        print('import error')
        pass
    else:
        print('initializing')
        colorama.init()
        print('initialized')

Seems silly, I know, but that would tell us exactly what your machine is doing.

[acastaner]

Doesn't seem silly at all, I'll try this and let you know.

[acastaner]

Tried the code snipped, and colorama seems to initialize fine?

E:\Music\blues\Muddy Waters>beet import "1989 - The Chess Box"
importing
imported
initializing
initialized

But then the rest of the import still shows the color control sequences.

[sampsyo]

Extremely mysterious! I took a look at the issues for Colorama and it sounds like there may be a problem with Colorama on Python 3 and writing raw bytes (which the only way we can control the output encoding). We probably have essentially the same issue as the one described here: https://github.com/tartley/colorama/issues/125#issuecomment-291140235

It looks like we'll need a different tactic. The problem, FWIW, is that this printing stanza: https://github.com/beetbox/beets/blob/6185c6ad4ba689ed438c2075699eb12b24ef5daf/beets/ui/init.py#L150-L152

is bypassing Colorama's translation by using sys.stdout.buffer to write bytes directly. Apparently, Colorama assumes that you only ever use sys.stdout directly. To control the encoding, we can't do that. (As an aside, I believe this is an important design flaw in Python 3—two steps forward, one step back.)

We'll need to look around for a way to coax Colorama into working on bytes on Python 3, in the same way as it does on Python 2. :confused: I'm not exactly sure how we'll manage this, but I'm marking this as a bug so we can look a little more closely.

[acastaner]

No problem, let me know if I can help further.

[RollingStar]

FWIW, I gave up on using cmd.exe and moved to Powershell.

[bleuge]

Same here, moved to PS, unusable in standard Windows 10 commandline.

[acastaner]

I'm running into issues when using PS. It somehow can't find the specified directories:

PS C:\Users\Arnaud\Desktop> beet import '.\Overkill - 1989 - The Years of Decay (Megaforce) V0' --set savedir="metal" error: no such file or directory: .\Overkill - 1989 - The Years of Decay (Megaforce) V0" --set savedir=metal

I'm not sure why it's doing this (and it's probably not related to beets) and that's why I tend to use cmd.exe instead.

[sampsyo]

Wow; that's pretty weird. It seems like single quotes should work as you'd expect… https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_quoting_rules

[jackwilsdon]

It sounds like it may be worth tracking tartley/colorama/issues/21, as that will fix this.

[sampsyo]

Good find, @jackwilsdon!

Yes, it does look like we either need to wait for that to get fixed, or we need to find a way to manually send bytes through the library.

[RollingStar]

FWIW, I can't get it to work anymore in Powershell either (maybe a Windows update broke it?). But it does work with Cmder Mini. http://cmder.net/

[ther0n]

You can fix this issue in cmd.exe using ANSICON. Still doesn't fix the issue in powershell though.

http://softkube.com/blog/ansi-command-line-colors-under-windows

[matlads]

Apparently using conEmu works well according to what this posted on discourse (see comment number 9):

https://discourse.beets.io/t/basic-usage/268/9

[tarrasque73]

I can confirm that I have the same issue using the command prompt in windows 10 but not using ConEmu.

[jtmoon79]

tl;dr same issue in powershell (not specific to cmd.exe), here's my setup.

Errant output

See the last part

PS> beet -v import ".\Quasi -- R&B Transmogrification"
no user configuration found at C:\Users\user1\AppData\Roaming\beets\config.yaml
data directory: C:\Users\user1\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: C:\Users\user1\AppData\Roaming\beets\library.db
library directory: C:\Users\user1\Music
Sending event: library_opened
Sending event: import_begin
state file could not be read: [Errno 2] No such file or directory: 'C:\\Users\\user1\\AppData\\Roaming\\beets\\state.pickle'
Sending event: import_task_created
Sending event: import_task_start
Looking up: C:\Temp\Quasi -- R&B Transmogrification
Tagging  -
No album ID found.
Search terms:  -
Album might be VA: True
Evaluating 0 candidates.

C:\Temp\Quasi -- R&B Transmogrification (14 items)
Sending event: before_choose_candidate
No matching release found for 14 tracks.
For help, see: http://beets.readthedocs.org/en/latest/faq.html#nomatch
[36;01m[S][39;49;00mkip, [34;01mU[39;49;00mse as-is, as [34;01mT[39;49;00mracks, [34;01mG[39;49;00mroup albums, [34;01mE[39;49;00mnter search, enter [34;01mI[39;49;00md, a[34;01mB[39;49;00mort?

Using

Software versions, other stuff

• python version 3.7.1

• PowerShell 5.1.17763.316

• Windows 10.0.17763.

• python libraries

PS> pipenv graph
beets==1.4.7
  - colorama [required: Any, installed: 0.4.1]
  - jellyfish [required: Any, installed: 0.7.1]
  - munkres [required: Any, installed: 1.1.2]
  - musicbrainzngs [required: >=0.4, installed: 0.6]
  - mutagen [required: >=1.33, installed: 1.42.0]
  - pyyaml [required: Any, installed: 5.1]
  - six [required: >=1.9, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.0.23]

• other tests

PS> python -c 'import sys; print(sys.platform)'
win32

PS> python -c 'import colorama; colorama.init()'

(no error).

---

Possibly related to tartley/colorama/issues/48 (bettter captured by tartley/colorama/issues/79). Very likely other tartley/colorama issues, too.

[tman144566]

I resolved this issue by enabling ANSI Terminal Control in the registry. I used Glenn Slayden solution. https://superuser.com/questions/413073/windows-console-with-ansi-colors-handling

[Kfftfuftur]

The problem seems to be in the print_ function that is beeing used to avoid exceptions by the default print function.

On windows we rely on colorama to convert ANSI color codes to win32 calls during encoding. Encoding the string directly seems to bypass that and break colorized output.

simply writing the text to stdout or even just passing it to the default print function would fix colors on windows however that would also throw encoding errors and break the encoding overwrite described in that function.

[sampsyo]

Yep, that’s a perfect summary. This thing about exceptions is still a sort of surprising downside of the Python 3 transition.

[lazka]

Windows ANSI support can also be enabled at runtime: https://github.com/mesonbuild/meson/blob/c53b6379597be5961b4e69e7f187608452874e4c/mesonbuild/mlog.py#L28

Maybe an alternative to colorama?

[JohnKara]

Is there a specific reason we use sys.stdout.buffer.write instead of sys.stdout.write? Because we could just decode the "out" variable and then use sys.stdout.write after we have replaced the unwanted characters using the encode() function. The below code works for both cmd and PS:

if hasattr(sys.stdout, 'buffer'):
    out = txt.encode(_out_encoding(), 'replace')
    txt = out.decode()
    sys.stdout.write(txt)

(I'm new to this project so forgive me if I missed something)

[sampsyo]

Good question; yes! The problem lies in our need to write raw bytes to stdout. The most prominent place where this comes up is when printing filesystem paths. On Unix, these are byte strings, not Unicode strings in any particular encoding. The stdout stream on Unix is also made of raw bytes, so we should be able to print them in “pristine” condition.

However, sys.stdout itself is a Unicode stream. Printing a path to it, depending on the system’s configured locale, might crash. Using the replace policy when round-tripping through Unicode is lossy, so paths might not be preserved. This make it impossible to do, for example, stuff like beet ls -p | xargs rm that consumes filenames on stdout.

Relatedly, the only way we currently have to let the beets configuration override the system’s locale is to write bytes directly to the underlying buffer.

(This is neither here nor there, but I think this whole I/O encoding business was and remains a design flaw in Python 3. It’s the one and only thing I still miss from the Python 2 days.)

[minchinweb]

@sampsyo

On Unix, these are byte strings, not Unicode strings in any particular encoding. The stdout stream on Unix is also made of raw bytes, so we should be able to print them in “pristine” condition.

Since we are trying to get colour output on Windows, could we just swap where to print: Use the buffer directly on Unix, and Python's print on Windows?

[sampsyo]

That seems like a great idea to at least try! I can imagine something going wrong with Windows' special treatment of UTF-16, but it could totally work. If somebody has access to a Windows machine and is willing to give it a shot, I would be super interested.

[minchinweb]

@sampsyo : I have a Windows machine, and I'd be happy to test! (I don't think I have the time to write the code myself at the moment though...)

Windows has gotten a lot better with handling UTF with the newest versions of PowerShell and Windows Terminal.

[jyrg02]

Hey I had a similar issue and switching to ConEMu worked to solve my issue.