beets
beets copied to clipboard
`beet import` with `lyrics` required incredible time
Problem
When I'm importing some rare music beet import stucks huge time.
Running this command in verbose (-vv) mode:
beet -vv import ~/Downloads/archive_mp3_1077.zip
user configuration: /Users/catap/.config/beets/config.yaml
data directory: /Users/catap/.config/beets
plugin paths:
Sending event: pluginload
library database: /Users/catap/.config/beets/library.db
library directory: /Volumes/music
Sending event: library_opened
Sending event: import_begin
Extracting archive: /Users/catap/Downloads/archive_mp3_1077.zip
Archive extracted to: /var/folders/rq/wqy42lq543z9rwrlj7vf1h4r0000gn/T/tmp8heg17sp
Sending event: import_task_created
Sending event: import_task_start
Looking up: /var/folders/rq/wqy42lq543z9rwrlj7vf1h4r0000gn/T/tmp8heg17sp
Tagging Михаил Щербаков - Минские встречи, часть 1. 1988 - 1989
No album ID found.
Search terms: Михаил Щербаков - Минские встречи, часть 1. 1988 - 1989
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'минские встречи, часть 1. 1988 - 1989', 'artist': 'михаил щербаков', 'tracks': '35'}
Requesting MusicBrainz release bf9cd789-af42-4566-8173-42ea7d48b4cb
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые (bf9cd789-af42-4566-8173-42ea7d48b4cb)
Computing track assignment...
...done.
Success. Distance: 0.02
Requesting MusicBrainz release 06e8cf68-edbd-4924-9756-4ffa1838dc87
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Михаил Щербаков - Минские встречи, часть 3. Двухтысячные (06e8cf68-edbd-4924-9756-4ffa1838dc87)
Computing track assignment...
...done.
Success. Distance: 0.53
Requesting MusicBrainz release f9bcf22f-c20b-4aec-bef2-ba5f6c238ed8
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Михаил Щербаков - Минские встречи, часть 2. Девяностые (f9bcf22f-c20b-4aec-bef2-ba5f6c238ed8)
Computing track assignment...
...done.
Success. Distance: 0.49
Requesting MusicBrainz release d9a3ac8a-2a02-40e2-a027-5e875bcf802c
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Михаил Щербаков - Избранное, Часть 1 (d9a3ac8a-2a02-40e2-a027-5e875bcf802c)
Computing track assignment...
...done.
Success. Distance: 0.65
Requesting MusicBrainz release 3acf75e7-2257-4cb9-abba-e684df232ae2
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Михаил Щербаков - Избранное, Часть 2 (3acf75e7-2257-4cb9-abba-e684df232ae2)
Computing track assignment...
...done.
Success. Distance: 0.57
Evaluating 5 candidates.
/var/folders/rq/wqy42lq543z9rwrlj7vf1h4r0000gn/T/tmp8heg17sp (35 items)
Sending event: import_task_before_choice
Sending event: before_choose_candidate
Correcting tags from:
Михаил Щербаков - Минские встречи, часть 1. 1988 - 1989
To:
Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые
URL:
https://musicbrainz.org/release/bf9cd789-af42-4566-8173-42ea7d48b4cb
(Similarity: 98.5%) (album) (CD)
Sending event: import_task_choice
Sending event: import_task_apply
0 of 35 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
smartplaylist: _Everything.m3u will be updated because of Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Вступление
smartplaylist: $albumartist/$album%aunique{}.m3u will be updated because of Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Вступление
smartplaylist: $albumartist/_Everything.m3u will be updated because of Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Вступление
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
fetchart: trying source filesystem for album Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые
fetchart: trying source coverart for album Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые
fetchart: getting URL: https://coverartarchive.org/release/bf9cd789-af42-4566-8173-42ea7d48b4cb
fetchart: downloading image: http://coverartarchive.org/release/bf9cd789-af42-4566-8173-42ea7d48b4cb/30693706160.jpg
fetchart: downloaded art to: /var/folders/rq/wqy42lq543z9rwrlj7vf1h4r0000gn/T/tmp23xx3iwy.jpg
fetchart: using remote image /var/folders/rq/wqy42lq543z9rwrlj7vf1h4r0000gn/T/tmp23xx3iwy.jpg
mbcollection: Updating MusicBrainz collection 04540b29-ba5e-4e0e-82e8-161035854aa4...
mbcollection: ...MusicBrainz collection updated.
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/%D0%9C%D0%B8%D1%85%D0%B0%D0%B8%D0%BB-%D0%A9%D0%B5%D1%80%D0%B1%D0%B0%D0%BA%D0%BE%D0%B2/%D0%92%D1%81%D1%82%D1%83%D0%BF%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5 (404)
lyrics: Genius failed to find a matching artist for 'Михаил Щербаков'
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Вступление
Sending event: database_change
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/%D0%9C%D0%B8%D1%85%D0%B0%D0%B8%D0%BB-%D0%A9%D0%B5%D1%80%D0%B1%D0%B0%D0%BA%D0%BE%D0%B2/%D0%92%D0%BE%D1%82-%D0%A2%D0%B0%D0%BA-%D0%9F%D1%80%D0%BE%D0%BF%D0%B5%D0%BB (404)
lyrics: Genius failed to find a matching artist for 'Михаил Щербаков'
lyrics: failed to fetch: http://www.tekstowo.pl/piosenka,_1052_1080_1090_1103___1060_1086_1084_1080_1085__1,_1042_1086_1090___1090_1072_1082___1074_1086_1090___1103___1083_1102_1073_1083_1102___1090_1077_1073.html (500)
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Scherbakov%2C-Mikhail/%D0%92%D0%BE%D1%82-%D0%A2%D0%B0%D0%BA-%D0%9F%D1%80%D0%BE%D0%BF%D0%B5%D0%BB (404)
lyrics: Genius failed to find a matching artist for 'Scherbakov, Mikhail'
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Вот так пропел
Sending event: database_change
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/%D0%9C%D0%B8%D1%85%D0%B0%D0%B8%D0%BB-%D0%A9%D0%B5%D1%80%D0%B1%D0%B0%D0%BA%D0%BE%D0%B2/%D0%9E%D0%B9%2C-%D0%94%D0%B0-%D0%A7%D1%82%D0%BE-%D0%96-%D0%AD%D1%82%D0%BE-%D0%A2%D0%B0%D0%BA%D0%BE%D0%B5 (404)
lyrics: Genius failed to find a matching artist for 'Михаил Щербаков'
lyrics: failed to fetch: http://www.tekstowo.pl/piosenka,wlodzimierz_wysocki,_1053_1072_1087_1088_1072_1089_1085_1086___1103___1083_1080_1094_1086___1089_1074_1086_1077___1088_1.html (500)
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Scherbakov%2C-Mikhail/%D0%9E%D0%B9%2C-%D0%94%D0%B0-%D0%A7%D1%82%D0%BE-%D0%96-%D0%AD%D1%82%D0%BE-%D0%A2%D0%B0%D0%BA%D0%BE%D0%B5 (404)
lyrics: Genius failed to find a matching artist for 'Scherbakov, Mikhail'
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: Михаил Щербаков - Минские встречи, часть 1. Восьмидесятые - Ой, да что ж это такое
Sending event: database_change
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/%D0%9C%D0%B8%D1%85%D0%B0%D0%B8%D0%BB-%D0%A9%D0%B5%D1%80%D0%B1%D0%B0%D0%BA%D0%BE%D0%B2/%D0%92%D1%81%D1%91-%D0%9A%D1%83%D0%B4%D0%B0-%D0%A2%D0%BE-%D0%A0%D0%B2%D0%B0%D0%BB%D1%81%D1%8F (404)
lyrics: Genius failed to find a matching artist for 'Михаил Щербаков'
^C^CTraceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/util/pipeline.py", line 423, in run_parallel
threads[-1].join(1)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1057, in join
self._wait_for_tstate_lock(timeout=max(timeout, 0))
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1069, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/ui/__init__.py", line 1280, in main
_raw_main(args)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/ui/__init__.py", line 1267, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/ui/commands.py", line 973, in import_func
import_files(lib, paths, query)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/ui/commands.py", line 943, in import_files
session.run()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/importer.py", line 341, in run
pl.run_parallel(QUEUE_SIZE)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/beets/util/pipeline.py", line 436, in run_parallel
thread.join()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1053, in join
self._wait_for_tstate_lock()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1069, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py'>
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1448, in _shutdown
lock.acquire()
KeyboardInterrupt:
Led to this problem: that I've interrupted it after 15 minutes of waiting.
Setup
- OS: macOS 11.6
- Python version: 3.9.7
- beets version: 1a5ddf89b5fe7191edb085bf38f9e0f3f434b1a8
- Turning off plugins made problem go away (yes/no): yes, without
lyricsplugin everything works very fast.
My configuration (output of beet config) is:
lyrics:
bing_lang_from: []
auto: yes
sources: musixmatch genius tekstowo google
google_API_key: REDACTED
google_engine_ID: REDACTED
bing_client_secret: REDACTED
bing_lang_to:
genius_api_key: REDACTED
fallback:
force: no
local: no
directory: /Volumes/music
asciify_paths: yes
import:
move: yes
follow:
auto: yes
email: REDACTED
password: REDACTED
userid: REDACTED
fetchart:
sources: filesystem coverart itunes amazon albumart lastfm
lastfm_key: REDACTED
auto: yes
minwidth: 0
maxwidth: 0
quality: 0
max_filesize: 0
enforce_ratio: no
cautious: no
cover_names:
- cover
- front
- art
- album
- folder
google_key: REDACTED
google_engine: 001442825323518660753:hrh5ch1gjzm
fanarttv_key: REDACTED
store_source: no
high_resolution: no
musicbrainz:
user: catap
pass: REDACTED
mbcollection:
auto: yes
remove: yes
collection: 04540b29-ba5e-4e0e-82e8-161035854aa4
smartplaylist:
auto: yes
relative_to: /Volumes/music
prefix: REDACTED
urlencode: yes
forward_slash: yes
playlist_dir: ~/Music/playlists
playlists: [{name: _Everything.m3u, query: ''}, {name: $albumartist/_Everything.m3u, query: ''}, {name: '$albumartist/$album%aunique{}.m3u', query: ''}]
convert:
copy_album_art: yes
auto: no
format: flac
formats:
copy_m4a:
command: ffmpeg -i $source -y -vn -c:a copy $dest
extension: m4a
aac:
command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
extension: m4a
alac:
command: ffmpeg -i $source -y -vn -acodec alac $dest
extension: m4a
flac: ffmpeg -i $source -y -vn -acodec flac $dest
mp3: ffmpeg -i $source -y -vn -aq 2 $dest
opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
dest:
pretend: no
link: no
hardlink: no
threads: 12
id3v23: inherit
max_bitrate: 500
tmpdir:
quiet: no
embed: yes
paths: {}
no_convert: ''
never_convert_lossy_files: no
album_art_maxwidth: 0
delete_originals: no
paths:
default: '%the{$albumartist}/$album%aunique{}/$track $title'
singleton: Non-Album/$artist/$title
comp: Compilations/$album%aunique{}/$track $title
ibroadcast:
username: [email protected]
password: jyhXyk-capzuh-2xunvi
auto: no
plugins: convert duplicates embedart fetchart follow fuzzy ibroadcast info lastgenre lyrics mbcollection mbsubmit mbsync missing smartplaylist the web
embedart:
maxwidth: 0
auto: yes
compare_threshold: 0
ifempty: no
remove_art_file: no
quality: 0
fuzzy:
prefix: '~'
threshold: 0.7
the:
the: yes
a: yes
format: '{0}, {1}'
strip: no
patterns: []
duplicates:
album: no
checksum: ''
copy: ''
count: no
delete: no
format: ''
full: no
keys: []
merge: no
move: ''
path: no
tiebreak: {}
strict: no
tag: ''
lastgenre:
whitelist: yes
min_weight: 10
count: 1
fallback:
canonical: no
source: album
force: yes
auto: yes
separator: ', '
prefer_specific: no
title_case: yes
mbsubmit:
format: $track. $title - $artist ($length)
threshold: medium
missing:
count: no
total: no
album: no
web:
host: 127.0.0.1
port: 8337
cors: ''
cors_supports_credentials: no
reverse_proxy: no
include_paths: no
readonly: yes
Ok, the root cause of issue is tekstowo. As soon as I disabled it, everything seems ok again.
Hm, I can't open https://www.tekstowo.pl nor http://www.tekstowo.pl ... is it dead?
It had opened, but it opens more than a minute per page, well, I guess I understand why it is so slow.
Ah, that's too bad! Maybe we should add a timeout to avoid mysterious stalls like this… any interest in attempting to hack that on?
@sampsyo I not sure that adding timeout is good idea. Adding small timeout won't help, and huge... well... :)
I feel that an issue with this web site (or my internet?). I think so because response time quite unstable:
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92666 0 92666 0 0 1892 0 --:--:-- 0:00:48 --:--:-- 25826
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92666 0 92666 0 0 49730 0 --:--:-- 0:00:01 --:--:-- 50008
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92666 0 92666 0 0 275k 0 --:--:-- --:--:-- --:--:-- 284k
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92666 0 92666 0 0 1619 0 --:--:-- 0:00:57 --:--:-- 24112
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1040 100 1040 0 0 127 0 0:00:08 0:00:08 --:--:-- 263
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92666 0 92666 0 0 292k 0 --:--:-- --:--:-- --:--:-- 303k
√ ~ % curl https://www.tekstowo.pl -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1040 100 1040 0 0 137 0 0:00:07 0:00:07 --:--:-- 238
√ ~ %
After reading a bit code: seems that it is a good idea to add timeouts for all requests.* calls.