beets icon indicating copy to clipboard operation
beets copied to clipboard

Published 20 hours ago • verified publisher icon beetbox

'charmap' codec can't encode character '\u0301' in position 79: character maps to <undefined>

Open rombat opened this issue 3 years ago • 2 comments

import crashin when importing Russian Circles last album Gnosis. Considering the error message, it's probably due to track 5 title, Ó Braonáin

Problem

Running this command in verbose (-vv) mode:

beet -vv import '.\2022 - Gnosis (24-96)\'
user configuration: C:\Users\tinbapakk\AppData\Roaming\beets\config.yaml
data directory: C:\Users\tinbapakk\AppData\Roaming\beets
plugin paths:
inline: adding item field disc_and_track
artresizer: method is (2, (7, 1, 0), False)
lyrics: Disabling google source: no API key configured.
Sending event: pluginload
library database: D:\Downloads\beets\musiclibrary.db
library directory: D:\Downloads\beets
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: D:\Downloads\soulseektemp\2022 - Gnosis (24-96)
Tagging Russian Circles - Gnosis
No album ID found.
Search terms: Russian Circles - Gnosis
Additional search terms: {'year': 2022, 'catalognum': ''}
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'gnosis', 'artist': 'russian circles', 'tracks': '7', 'date': '2022'}
Requesting MusicBrainz release 47409bfe-a2ec-43c5-85d2-afec8ac55f33
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Russian Circles - Gnosis (47409bfe-a2ec-43c5-85d2-afec8ac55f33)
Computing track assignment...
...done.
Success. Distance: 0.02
Requesting MusicBrainz release 5fdaf376-1bdf-4d38-8a87-a0da2fa7a81c
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Russian Circles - Gnosis (5fdaf376-1bdf-4d38-8a87-a0da2fa7a81c)
Computing track assignment...
...done.
Success. Distance: 0.05
Requesting MusicBrainz release acac3cdb-ccd5-3736-87b3-1554de5a83be
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Russian Circles - Geneva (acac3cdb-ccd5-3736-87b3-1554de5a83be)
Computing track assignment...
...done.
Success. Distance: 0.58
Requesting MusicBrainz release b3b84abd-cf73-4b98-9285-61ab85b5bbf7
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Russian Circles - Station (b3b84abd-cf73-4b98-9285-61ab85b5bbf7)
Computing track assignment...
...done.
Success. Distance: 0.54
Requesting MusicBrainz release faea60b5-152a-4714-b80a-c5441a115750
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Russian Circles - Geneva (faea60b5-152a-4714-b80a-c5441a115750)
Computing track assignment...
...done.
Success. Distance: 0.61
bandcamp: Searching as for Gnosis using {'query': 'Gnosis', 'artist': 'Russian Circles', 'label': '', 'search_type': 'a'}
Sending event: albuminfo_received
Candidate: Russian Circles - Gnosis (https://russiancircles.bandcamp.com/album/gnosis)
Computing track assignment...
...done.
Success. Distance: 0.07
Sending event: albuminfo_received
Candidate: Aethyrick - Gnosis (https://aethyrick.bandcamp.com/album/gnosis)
Computing track assignment...
...done.
Success. Distance: 0.60
Sending event: albuminfo_received
Candidate: Aethyrick - Gnosis (https://aethyrick.bandcamp.com/album/gnosis#p3788263956)
Computing track assignment...
...done.
Success. Distance: 0.57
Sending event: albuminfo_received
Candidate: Aethyrick - Gnosis (https://aethyrick.bandcamp.com/album/gnosis#p2563406104)
Computing track assignment...
...done.
Success. Distance: 0.62
Evaluating 9 candidates.

D:\Downloads\soulseektemp\2022 - Gnosis (24-96) (7 items)
Sending event: import_task_before_choice
Sending event: before_choose_candidate
Tagging:
    Russian Circles - Gnosis
URL:
    https://musicbrainz.org/release/47409bfe-a2ec-43c5-85d2-afec8ac55f33
(Similarity: 97.7%) (media) (Digital Media, 2022, XW, Sargent House)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? A
Sending event: import_task_choice
Sending event: import_task_apply
0 of 7 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 01 Tupilak.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 02 Conduit.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 03 Gnosis.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 05 Ó Braonáin.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 06 Betrayal.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 07 Bloom.flac
convert: Encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 04 Vlastimil.flac
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 04 Vlastimil.flac
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 07 Bloom.flac
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: after_write
Sending event: after_write
Sending event: database_change
Sending event: database_change
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 06 Betrayal.flac
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 01 Tupilak.flac
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 02 Conduit.flac
convert: Finished encoding D:\Downloads\soulseektemp\2022 - Gnosis (24-96)\Russian Circles - Gnosis - 03 Gnosis.flac
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: write
zero: albumartist_sort: Russian Circles -> None
zero: composer_sort:  -> None
zero: artist_sort: Russian Circles -> None
zero: original_month: 8 -> None
zero: original_day: 19 -> None
Sending event: after_write
Sending event: after_write
Sending event: after_write
Sending event: after_write
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Traceback (most recent call last):
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\Scripts\beet-script.py", line 33, in <module>
    sys.exit(load_entry_point('beets==1.6.0', 'console_scripts', 'beet')())
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\ui\__init__.py", line 1285, in main
    _raw_main(args)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\ui\__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\ui\commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\ui\commands.py", line 943, in import_files
    session.run()
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\util\pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\util\pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\util\pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\importer.py", line 1535, in plugin_stage
    func(session, task)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\plugins.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beetsplug\convert.py", line 183, in auto_convert
    par_map(lambda item: self.convert_on_import(config.lib, item),
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beets\util\__init__.py", line 1060, in par_map
    pool.map(transform, items)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 774, in get
    raise self._value
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beetsplug\convert.py", line 183, in <lambda>
    par_map(lambda item: self.convert_on_import(config.lib, item),
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beetsplug\convert.py", line 513, in convert_on_import
    self.encode(command, item.path, dest)
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\site-packages\beetsplug\convert.py", line 216, in encode
    encode_cmd.append(args[i].encode(util.arg_encoding()))
  File "C:\Users\tinbapakk\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u0301' in position 79: character maps to <undefined>

and it removes the original flac file by the way

Setup

  • OS: Windows 11 beets version 1.6.0 Python version 3.10.6 plugins: bandcamp, convert, embedart, fetchart, ftintitle, inline, lyrics, zero

My configuration (output of beet config) is:

directory: D:\Downloads\beets
library: D:\Downloads\beets\musiclibrary.db

import:
    copy: yes
    write: yes
    timid: yes
    languages: fr en jp

artist_credit: yes
per_disc_numbering: yes
original_date: yes

match:
    preferred:
        media: ['CD', 'Digital Media|File']
        original_year: yes

paths:
    default: $albumartist/$year - $album%aunique{}/$disc_and_track - $title
    singleton: Non-Album/$artist/$title
    albumtype:soundtrack: Soundtracks/$album/$track - $title
    albumtype:anime: Anime/[$original_year-$original_month-$original_day] $album/$track - $title
    albumtype:game: Game/[$original_year-$original_month-$original_day] $album/$track - $title
    comp: Various Artists/[$original_year-$original_month-$original_day] $album%aunique{}/$track - $title

musicbrainz:
    extra_tags: [year, catalognum]
    
plugins: fetchart lyrics embedart inline convert zero ftintitle bandcamp

fetchart: 
    cover_format: JPEG
    minwidth: 800
    maxwidth: 1600
    sources: filesystem coverart itunes amazon albumart fanarttv lastfm google
#    quality: 90
    
embedart:
    maxwidth: 1200
#    quality: 90
    
item_fields:
    disc_and_track: u'%2i.%02i' % (disc, track) if disctotal > 1 else u'%02i' % (track)
    
convert:
    format: aac
    formats: 
        aac:
            command: qaac --ignorelength --threading -q 2 --tvbr 118 --rate keep $source -o $dest
            extension: m4a
    dest: D:\Downloads\beets
    copy_album_art: yes
    embed: yes
    auto: yes
    never_convert_lossy_files: yes

zero:
    auto: yes
    fields: albumartist_sort composer_sort artist_sort original_month original_day
    
ftintitle:
    auto: yes
        
bandcamp:
    art: yes
    genre:
        maximum: 1

lyrics:
    auto: true,
    sources: google genius musixmatch

rombat avatar Aug 23 '22 10:08 rombat

That's always quite frustrating! The issue here is that beets is trying to invoke a command-line tool as part of the convert plugin, but to do so, it needs to construct a command line using the filename of the file to convert. However, that filename is not representable in the system's configured encoding for command-line arguments!

It's not entirely clear what beets should do here, although crashing is probably not the thing. Perhaps it's possible, on Windows, to construct an all-Unicode (using Windows' weird variant of UTF-16) command line, in which case we should do that. Otherwise, maybe we should just catch this error and report it before moving on.

As a short-term fix, you might consider seeing whether it is possible to reconfigure Windows to use a modern Unicode encoding for arguments. Unfortunately, I'm not too familiar with Windows, so I'm not sure where to point you…

sampsyo avatar Aug 23 '22 20:08 sampsyo

Thanks for the input ! For anyone wondering, I've made it work (with kanjis, not the same file but I guess it should do too), thanks to a beta feature in Windows to set the system locale in UTF-8. I've not yet noticed any particular problem (due to its beta status) so far.

Edit: Actually it does mess up with some stuff (I got some broken buttons in MusicBee for instance). I'll try something else.

rombat avatar Aug 24 '22 04:08 rombat

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 29 '22 00:10 stale[bot]