pyacoustid
pyacoustid copied to clipboard
beetbox
acoustid submission error: response is not valid JSON - 504 gateway time-out
I'm adding new releases in MB database and I was curious after tagging why the fingerprint wasn't on the website and I received no errors so I explicitly tried with the submit comment.
Output:
$ beet -vvv submit "./"
user configuration: /home/user/.config/beets/config.yaml
data directory: /home/user/.config/beets
plugin paths: /home/user/.beets-fetchartist
/home/user/.local/lib/python3.8/site-packages/beets/mediafile.py:21: UserWarning: beets.mediafile is deprecated; use mediafile instead
warnings.warn("beets.mediafile is deprecated; use mediafile instead")
Sending event: pluginload
inline: adding item field my_samplerate
inline: adding item field my_bitrate
inline: adding item field is_flac
inline: adding item field is_album
inline: adding item field multidisc
inline: adding item field vinyl
inline: adding item field compilation_display
inline: adding album field fileformat
inline: adding album field av_bitrate
inline: adding album field avg_samplerate
inline: adding album field avg_bitdepth
library database: /media/music/.beet_library.blb
library directory: /media/Nas/Music
Sending event: library_opened
chroma: /media/Nas/Music/KNEECAP/Singles/2017 - C.E.A.R.T.A [MP3 320]/01 - C.E.A.R.T.A.mp3: fingerprint exists, skipping
chroma: submitting MBID
chroma: submitting 1 fingerprints
chroma: acoustid submission error: response is not valid JSON
Sending event: cli_exit
I tried multiple tracks, I always get that error - I also confirmed my API key.
Config:
original_date: no
per_disc_numbering: yes
threaded: yes
clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", ".pls"]
ignore: ["*~", "System Volume Information"]
import:
languages: en de fr jp
move: yes
copy: no
bell: yes
ui:
color: yes
colors:
text_success: green
text_warning: yellow
text_error: red
text_highlight: red
text_highlight_minor: lightgray
action_default: turquoise
action: cyan
pluginpath:
~/.beets-fetchartist
plugins: inline convert bpm discogs lastgenre absubmit acousticbrainz scrub fromfilename chroma acousticbrainz beatport the ftintitle fetchart embedart extrafiles edit fetchartist
aunique:
keys: albumartist album year
disambiguators: albumtype year country label catalognum albumdisambig releasegroupdisambig
bracket: '()'
set_fields:
how: ''
item_fields:
my_samplerate: str(round(samplerate / 1000))
my_bitrate: str(round(bitrate/1000))
is_flac: 1 if format == "FLAC" else 0
is_album: 1 if albumtype.upper() == "ALBUM" else 0
multidisc: 1 if disctotal > 1 else 0
vinyl: 1 if media in ['12" Vinyl', 'Vinyl'] else 0
compilation_display: '''Soundtracks'' if albumtype.upper() == ''SOUNDTRACK'' else ''Compilations'''
#first_artist: albumartist.split(', ',1)[0:1][0]
album_fields:
fileformat: |
formatList = []
for item in items:
formatList.append(item.format.upper())
return formatList
av_bitrate: |
total = 0
for item in items:
total += item.bitrate
return str(round(total / len(items) / 1000))
avg_samplerate: |
total = 0
for item in items:
total += item.samplerate
return str(round(total / len(items) / 1000))
avg_bitdepth: |
total = 0
for item in items:
total += item.bitdepth
return str(round(total / len(items)))
paths:
#default: %upper{%left{%asciify{%the{$albumartist}},1}}/$albumartist/%if{$year,$year - ,%if{$original_year,$original_year - }}$album%aunique{} ($format)/%if{$multidisc,$disc-}$track - $title
# Albums/A/ASCI Artist Name, The/[YEAR] ASCI Album Name, The [EP]/01 - Track Name.mp3
# %bucket{%upper{%left{%the{$albumartist},1}}}/
default: '%the{$albumartist}/%if{$original_year,$original_year,%if{$year,$year,0000}} - $album%aunique{} [%ifdef{how,$how }%if{$vinyl,Vinyl }%upper{$format}%if{$is_flac, $avg_samplerate-$avg_bitdepth, $av_bitrate}]/%if{$multidisc,$disc-}$track - $title'
albumtype:single: '%the{$albumartist}/Singles/%if{$original_year,$original_year,%if{$year,$year,0000}} - $album%aunique{} [%ifdef{how,$how }%if{$vinyl,Vinyl }%upper{$format}%if{$is_flac, $avg_samplerate-$my_samplerate, $my_bitrate}]/%if{$multidisc,$disc-}$track - $title'
albumtype:ep: '%the{$albumartist}/EPs/%if{$original_year,$original_year,%if{$year,$year,0000}} - $album%aunique{} [%ifdef{how,$how }%if{$vinyl,Vinyl }%upper{$format}%if{$is_flac, $avg_samplerate-$avg_bitdepth, $av_bitrate}]/%if{$multidisc,$disc-}$track - $title'
albumtype:soundtrack: Soundtracks/$album%aunique{}%if{$year, ($year),%if{$original_year, ($original_year)}} [%ifdef{how,$how }%if{$vinyl,Vinyl }%upper{$format}%if{$is_flac, $avg_samplerate-$avg_bitdepth, $av_bitrate}]/%if{$vinyl,$track_alt,%if{$multidisc,$disc-}$track} - $title
comp: $compilation_display/%if{$year,$year - ,%if{$original_year,$original_year - }}$album%aunique{} [%ifdef{how,$how }%upper{$format}%if{$is_flac, $avg_samplerate-$avg_bitdepth, $av_bitrate}]/%if{$multidisc,$disc-}$track - $title
singleton: '%the{$artist}/Non-Album/$title%if{$original_year, ($original_year),%if{$year, ($year),}} [%ifdef{how,$how }%upper{$format}%if{$is_flac, $samplerate-$bitdepth, $my_bitrate}]'
replace:
'[\\/]': _
'^\.': _
'[\x00-\x1f]': _
'[<>:"\?\*\|]': _
'\.$': _
'\s+$': ''
'^\s+': ''
'^-': _
'[\u2018\u2019]': ''''
'[\u201c\u201d\u2033]': '"'
check:
import: yes
write-check: yes
write-update: yes
convert-update: yes
threads: 4
ftintitle:
auto: yes
drop: no
format: feat. {0}
lyrics:
fallback: ''
google_engine_ID: xxxxxxx:xxxxxxx
google_API_key: xxxxxxx-xxxxxxx
lastgenre:
min_weight: 80
count: 5
prefer_specific: no
separator: '; '
extrafiles:
patterns:
artworkdir:
- '[sS]cans/'
- '[aA]rtwork/'
- '[aA]rtworks/'
- '[cC]overs/'
lyrics:
- '*.lrc'
lyrics2:
- 'Lyrics/'
nfo:
- 'album.nfo'
paths:
artworkdir: $albumpath/artwork
lyrics: $albumpath/$filename.lrc
lyrics2: $albumpath/Lyrics
embedart:
# I dont want album art auto-embedded either
auto: no
fetchart:
auto: yes
minwidth: 400
maxwidth: 2000
sources: filesystem coverart itunes amazon albumart
fetchartist:
filename: "poster"
acousticbrainz:
force: yes
absubmit:
extractor: ~/streaming_extractor_music
auto: yes
acoustid:
apikey: xxxxxxx
chroma:
auto: yes
Any tips?
Hmm; that's odd. Any chance you could view the API request and response to see if there's an underlying error? Observing the traffic with Wireshark or similar would work, as would just printing out the response inside the Python library.
It's a gateway timeout it seems:
This is from trying to match a recording as well. Seems like the API is broken atm?
Oh maybe. Thanks @BNolet I just install wireshark, I forgot I could use that.
The library returned error wasn't very helpful then, maybe it should've said: Service unavailable.
It seems beet (and picard) uses only album/artist to find matches indeed. II can't find a status page for the apis, we'll have to wait.
Yeah! A PR here would be welcome if anyone has a moment to add a better error message for 5xx error codes.
Welp, Picard is working just fine so I'm going to use that to tag my library :) https://picard.musicbrainz.org/
@BNolet On my end I can tag using beet, I just can't use fingerprints or upload them - same as Picard ATM.
Using picard if I try to upload a fingerprint I see the request in the logs with no error but it never shows on the website and it usually shows the fingerprint on the website pretty fast.
@Extarys Ah maybe Picard is just working for matching recordings and not submissions then, sorry to hear that
I think this need further investigation.
This is Picard log:
D: 11:00:57,461 /usr/lib/picard/picard/acoustid/__init__._run_next_task:224: Starting fingerprint calculator '/usr/bin/fpcalc' '/media/NAS/Music/KNEECAP/Singles/2017 - C.E.A.R.T.A [MP3 320]/01 - C.E.A.R.T.A.mp3'
D: 11:01:52,669 /usr/lib/picard/picard/acoustid/manager.submit:95: AcoustID: submitting total of 1 fingerprints...
D: 11:01:52,670 /usr/lib/picard/picard/acoustid/manager._batch_submit:111: AcoustID: submitting batch of 1 fingerprints (0 remaining)...
D: 11:01:52,671 /usr/lib/picard/picard/webservice/__init__.post:470: POST-DATA 'user=MYAPIKEY&fingerprint.0=AQADtErERKGSJFPA4skUNOuO8Joi6NwRakfTLR--HLmeCP1yNFK2Qj--iTOeC_0h_nh29B2h3_hBVZWHfELdWMJH4kKqFPpRyqnwXPjRZByqJgtn_JC3o40PP0T742jyoOyO68KJ1kdTp7iYH8-O8JsCZwsP5pHwHT1XJPWRfjweo3YTPEefoeGJncOXrEd-4fnx-gm083iOh4efYs-hnqjyIh94XA-xH0W0b_gMMU2O44vyIy9-Ge_xpPhhSQ3qIb-CPwcjaPmLUzim_Wik4dKhH19DrEOo9YGtK4J7oUcppWjs49yQ8ULlHD32HbOcB0czBo6FPpiZ49fx7WieD3p4NJl89DOakDp2psO3XEF_NM8C6bpxJTReTDl7hE-O-LiikMdz9FMFhzrxhB8Ov0ifog8iiYoE64fjg_lwBTr-YgqtHJOaHLWILQ8eaNpwH_1x-MF97D3k50If9AelVUavBV3R8MfKRIeO4yfa8LBOXDwGP8cPccM3Mke2H0ePK7ge43FyMIm04w-aPELyBfmPUoFzLwmsUzLaaKiYJCOaHdeFSiwR5h-ubDE85XiZ4-4xH0-Ia8F_tG3xw-olfMvh4E7gG7WGSQ55XEefgdWOY94LLfmO6uiTwxVj_MeDG5rf4cqCHj9sfdj14yv6C_3gG5odMGMeiMuDPjzkBv-Jf5CTcC6a9MH7IUUf9BSaJvTBOQsR53iw41K-4GqOKUOPUNqUoKmLeDyeGSM1cegTQsekG9VnNNR0XCEjnAe9w8-gR2GNNw92-BnuB2cY3NEha1EaCUcTa5Tw5IPhp-gDfdJRIVpSNF-LBzzxmUVzRPqUopoEncEp4oGzGxcD3fCRH_4VvNjCHyH-h-gFhwOpHU1afMnRE38yQ-uN50oFTdqX4C96fEbjo1aWDDcvkDh-XNmD68FfXAvxp_ihazqOv-imybjwo_FxHuoFc_iDr8eF9-AZBn7wH2cIXRfGLZEhXw5uPsGDFz2m_PgymvhzHDyeK9ij4_iN6olxJjAfyMKDx7A1_DO2d8XVwJGWE-Jx5gK1o6t0PApkHpMdaKmIn8PxiUb-6Kj-wM8VpD_wQ8uPS0PPWJiUQ8fBP_h8PMOb5OhDPEJ16D0-hehp_DjRpEkOWfFxEj2a5-Bx5biE8mi-HF-xhxgfXbgX40fHZE-Cs0AOU0e9hQZt5MWe4CUhT-Hxfah2OM_xELeUw9zx4D-eFv6h3LhnwgxO6cKnoz_YLsYV4_WhEVMeBTrzCT_e6PiD-FfQhxx-_EgflMfHDJPuQ9t0HHwOPVxw5Ti-HNqD_yjVCz8efNmhXSoatbjwEV1h9FmOn1B3MHbQOMdxQg-uSDbi6-j-GG_yBP2PNwi7oHkdSJ8SXEds7Dly5rhy4smOQ0udotoS3EJ_WNFy9KFxCX_RZGkOVdMx-cPN40_AJEen7MUH-ZhydBqenPjxh7iOJmSV4RNVwHfxYvOJUuOD6nh59OCoHH_QnkUzBrXQ5MNbH49xZsdR60K7BvXQOMWlBz-ao9uPTzpyNR3-oR-aH08SpcZP_EifKBzU7RvyOcFPCUyeox051D_-B3mQTDruHn_w5seP00d0NdDJSccOnQueH2HVB_6qwOgfFKkTQoqiiAWuY9x46Fk2Y7KD3OhutOvxSRZS6dAHu8hvo4mO4zrqIY35CH_ADdoavbh0nGieYup4_Hh05A7xrxJ6Bo7zwPGRH4ePslDk88JLIzx4XOKxP8iXoI8EbQ9zfEPO4Tkeoxd-uComK9HxFh-p42jUHP2Rh02gR0eWKCt6vD2e4zzyMDnUhhmNdnyC8Dn-olfR9OC-0IiDH3tefIeOH02rDa1iXDm05_CPakS-40XXI_1xamijYz305-h5UPIOzVpwog-mqIxxHdfxU_gRLrrgOB8xNkXa49qEJn-gi2BEF1ZPwQ-mB4_44M-hPcGWhjp8BW0G_dAVJcePXkT3I08WNFl0MEe-HOUL8QyP-phZtCqF5tRxMQvOC7pxyhK-B2EY4pM-7EcaHj_E8PikDBlnXJzR54jMBN-Lv_iEJmYMbdTxvfgRiv3RPHhvPB0aF5MWp0fuQz9exgMzNFV0PDqO6sWU-MST48ed4x9nJNPhH8dXFw9-jNKOE-FK8Iee43K4YNKOJlqClznsZMVDfEsOnegf7EEvfNFhHn8EKUfzGnIzC2182Cne7Kje4_ihPD3-HHKH6k_wPDiPw2MQWjciOj2u4yZ44VEe7PLwo8_hbDyeKFDN4IG548W_YntwF2J2fLsIalYu_GgUDz8uYW3RjMxR9sQZfLSQJ8d_oemPn1SwpcYlOEfIo9d6SArxo_kx97iicsGFxoJaY8qYQ12k48HOcKgVqcPRK5iaHYoS_EE_NM-MNjqOvYf1oNoOPhlOBu8tPAm-1fif4D9-PEPD6OhvfFES_OCDyyJ2Hf8H7YKpxkKtBX2iw3zwo6KCZzU0Sn1hzZgqHd1w6sOfHM71gpFIPKPxpjjEkXmg8MePqxTxCN0v5PuCBz_yHD5KHtNDHppt4-CP58EafYSOFx9x5tgP4wwgShAjFAEGCgmEUcIQQJShgoEhFSGOAQIAAMwQwhgAgghFGEBEGQWAMgoYI5ABQgAgBBMKAWABIQopApUhFhBgAAFOAimBAUAYoIxRAjlhhBQKcCKQAIYJAADAQBggCDRACAOAMRYIQI0ATgmigAgAOOAIAosgYxgCwrgkiBAEIKWAMAIhQYxTQBlnBDCOEQQFAURAoowxgAiKBBFIMASEYoAAgBBAAABBBWCMAAQUMIABYKwgRhhgECAKAgAEMQgwIoQAnDkiDCCWAYQARAwYIIDBAFAhFFQIAIIEA0oLBJhDBgEjDAACGIAgMYIAAIBxRjkLlGMKCSCMIsBQAaQSxCFgEKIMOYOUIsAAxgwUQDBKJABCAGAYARQsQgRwFABlkGUAAEOdYQoAgAQQQDrGEGAEMeAIIAAwRgICBGBmhOMAqQSFE4oYoIAFgDEKjACQAAIYEIgB45QSRBDAgIECGAQMkAoJIIRgkiHAkRHSEQQBIEgwACxQiBogFBKOKUYtQ4YwAgAShAAEhDNGAUQkMVYBA4gwwkggCCBGAaUMUkIISBQBCgpFEJFCEGAEwAgRKhAQgBoohEAGCaoAAMAABIxRQkpkBAFCOEAEAEABYIgQxDlhkCQAASKIowYBYAQSAgHnlCACUCMpcSIAQQwhAAhAiBAMGQAoAUiYIowSkgjFgEDACSEoAAowYxBSBkmklEIEMEWVEk4KAARQAimHjCDQMAGIYcYAYQQwADCCDAHCAUCQUswZABhAQFBCmAECGuCMBUAwo5BRSgAAAEBAUQAQBAIoJxBiQhFjBMACgcMMk9ZAhAgQTgCDBDEIAQcMAUpAI7RFhAEgEGCGEEaMEMAYJBVTACiHmEJIAWGAEIgZZISATAABBAAoECCYERAJYAEzzBhBiFSMKEIFIcQBQQQACEAmBTECEEQUMMYIAhgSghAihBDCAGKAYdZAIAhhFikDqCDMAIaAIJBYoQA&duration.0=222&mbid.0=65161f21-ae51-48f4-84c0-2f7f53662081&client=v8pQ6oyB&clientversion=2.3.1.final0&format=json'
D: 11:01:52,678 /usr/lib/picard/picard/webservice/ratecontrol.get_delay_to_next_request:113: ('api.acoustid.org', 80): First request
D: 11:01:52,680 /usr/lib/picard/picard/webservice/ratecontrol.increment_requests:138: ('api.acoustid.org', 80): Incrementing requests to: 1
D: 11:01:53,156 /usr/lib/picard/picard/webservice/ratecontrol.decrement_requests:146: ('api.acoustid.org', 80): Decrementing requests to: 0
D: 11:01:53,156 /usr/lib/picard/picard/webservice/__init__._handle_reply:419: Received reply for http://api.acoustid.org:80/v2/submit: HTTP 200 (OK)
D: 11:01:53,159 /usr/lib/picard/picard/acoustid/manager._batch_submit_finished:142: AcoustID: 1 fingerprints successfully submitted
D: 11:01:53,160 /usr/lib/picard/picard/file.update:614: Updating file <MP3File '01 - C.E.A.R.T.A.mp3'>
D: 11:01:53,161 /usr/lib/picard/picard/acoustid/manager._batch_submit:100: AcoustID: submitted all fingerprints
D: 11:01:53,161 /usr/lib/picard/picard/webservice/ratecontrol._out_of_backoff:222: ('api.acoustid.org', 80): oobackoff; delay: 1000ms -> 500ms; slow start; window size 1.000 -> 2.000
Fingerprint was uploaded: https://musicbrainz.org/recording/65161f21-ae51-48f4-84c0-2f7f53662081/fingerprints
I think yesterday I didn't properly submit in Picard, that is why I thought it wasn't wokring. If it works in Picard but not in Beet, I think it can impact the number of fingerprints that will end up on the Brainz server. :disappointed:
I also get the same error here (504), but querying the api with the same parameters through browser/wget/curl works fine.
Correction: if I send the request as plain text through curl it works, if I send it gzipped as I think pyacoustid does I get a 504. But I could easily have gotten something wrong in these tests. The 504 might be a problem server side then. Anyone know who to talk to for that?
Further info: if I disable gzipping in acoustid.py everything seems to work correctly. My picard sends the request as plain text (urlencoded) and works.
@skapazzo Can you explain how to disable gzip in acoustid? I just use it straight in beet though and the beet plugin config doesn't show much options. Do I need to change this directly in the python file?
Yes, I just hacked acoustid.py. I forked pyacoustid and uploaded the changes I used for testing. You can see the diff here: https://github.com/beetbox/pyacoustid/compare/master...skapazzo:skapazzo-temporary Beware, this was just a hack for testing, no idea about side effects.
Oh wow, thanks for tracking that down! Seems like something is wrong on the server side with gzipped requests.
Paging @lalinsky in case he hasn't seen this thread yet. It also looks like there is a related server issue filed at https://github.com/acoustid/acoustid-server/issues/50.
Thanks guys. The service has been heavily overloaded during the last weeks, so I had to work on some frontend proxying and it's possible that some gzipped request support got broken. I'll have a look quickly.
