Issue: Picard crashes loading large collection of tracks with Lastfmng at 1.9gb.

[pumalife]

My intention: I have a large collection (17,000 tracks) of already tagged (all has musicbrainz ID) mp3's and i want to reload into picard and add last fm ng tags.

My System: Windows 7, intel core i5-2410m 2.3ghz (sandy bridge, 2011), 6gb RAM, regular Sata Hard Drive: Dell Inspiron N5110 laptop.

My Problem: Picard crashes, caused by lastfmng when loading thousands of tracks tracks. 4 test cases: 600, 2500, 5000, 10000 tracks. 600 and 2500 was successful, but 5k and 10k crashed. my breakpoint falls between 2500 and 5000 tracks (probably closer to 5000) It seems to crash when picard.exe reaches 1.9gb of memory usage. Note is still have 1-2.5gb of system memory free. (ive looked at disabling other features in picard eg. coverart, track relationships, release relationships, folksonomy but it doesnt affect the outcomes.)

For Reference: 'album info loaded' means: album in right hand pane no longer says [loading album information]

_______________ test case 1: 10k files _______________ 10k (700albums(musicbrainz id)/65gb) Files

round 1 - lastfmng plugin enabled move all 10k tracks into right pane successfully grouped in albums waiting to load. Picard Crashes: 10:06 min in - 0 album info loaded (memory: 1.9GB) memory usage: physical working set hit 1,917,040k, private bytes at 1,899,792k, then crashed. picard was able to load all the tracks from left into right hand pane, but NONE of the album info was loaded. pending requests (green downpointing arrow in status bar of picard) still has queue (it was processing items/queue was decreasing).

round 2 - lastfmng plugin disabled ~30 seconds in- first album info loaded (memory at: NA) (eg. i see a gold record) 13:30 min in- all album info loaded (memory at: <450mb) with plugin disabled, picard immediately starts identiying albums info. one at a time, down the list, evenly spaced out, until all are identified...10 min later

_______________ test case 2: 5k files _______________ 5k (352 albums/29gb) files, the first half of the 10k files in test case 1.

round 1 - plugin enabled loads all tracks into right pane successfully. pending requests is now 0 28 min in- first album info loaded (memory at: 1.5gb, then slowly starts rising.) Picard Crashes: 49 min in - with still only one album info loaded. (memory at: 1.9gb) memmory usage: physical working set hit 1,900,060k, then crashed.

round 2 - plugin disabled not tested.

_______________test case 3: 600 tracks - baseline test _______________ for stats and baseline. no crashing. 600 tracks (28albums/1.7gb) - my entire beatles collection.

round 1 - plugin enabled, using custom configuration, config.ini 5 min in- first album info loaded (memory at: 396mb) 9:30min in- all album info loaded (memory at: 396mb)

round 2 - plugin enabled, removed config.ini: using only default values 5min in- first album info loaded (memory at: 391mb) 9:26min in- all album info loaded (memory at: 391mb)

round 3 - plugin disabled ~5 seconds in- first album info loaded 1:08min in- all album info loaded (memory at: 160mb)!! :)

_______________test case 4: 2.5k _______________ 2500 tracks (191albums/12.5gb)

round 1 - plugin enabled <25 min in- first album info loaded (memory at: 1.1gb) 28 min in- 2nd + album info begins loading. speed has increased, 20 albums loaded. (memory at: 1.1gb) 49 min in- all album info loaded (memory at: 1.1gb)

round 3 - plugin disabled ~20 seconds in- first album info loaded (memory at: 200mb) 4:10 min in- all album info loaded (memory at: 200mb)

_______________summary _______________

1. the the huge increase in time to load the albums with lastfmng, vs lastfmng disabled is concerning.
2. weird that it keeps all albums at [loading album information] for so long with lastfmng. while with plugin disabled, it begins loading album info immediately (one at a time, down the list).
3. why is it crashing when the picard process hits memmory usage of 1.9gb?
4. is there anything to be done, relatively easy fix?
5. obviously 2500 tracks per load is pretty good, not a big deal in the scheme of things.

Statistics from testing: with LAST FM NG disabled picard loaded 2500 tracks, with an average of 1.10 seconds/track with LAST FM NG enabled picard loaded 2500 tracks, with an average of 0.10 seconds/track these per track averages seems to scale down relatively close. ie 600 tracks takes 0.95 and .11 respectively.

Thanks, Colin

[fdemmer]

hi, thanks for doing the tests...

however, my first thought was "why would you want to retag everything at once!? of course keeping all metadata for 17k tracks in memory is huge." ;)

anyway, i'll try to answer some of your questions:

1. the plugin does a lot more, than just picard alone. picard fetches album info from their own custom api written for exactly that usecase (or at least with that in mind). the plugin has to work with a more general purpose api and makes a call for the album, the artists of the album (one if just one, multiple if various artists) and again for each track to collect tags.

that simply takes time and produces some excess data and there lies one first clue to memory usage: to reduce time, it caches all requests. a second call to the same thing later does not make a second request (during that picard session; not closing the program)

about timing: picard alone: roughly 1-2 http requests = ~0.1 sec; plugin: request for artist, album, tracks = 10-20 http requests = ~1 sec ... seems reasonable (without actually testing myself).

check the log to see it working... (turn on debug for even more info).

2. the plugin uses picards own task queueing which is also used to queue http requests. an album is only displayed as loaded once the plugin is also finished with it. there might be something i can do about it waiting for all albums to finish. i observed that myself, but that requires digging into picard's queue code and it was never a priority for me, since i usually only tag a hand full of albums at once.

if you drop in the albums in smaller "groups", those "groups" should finish one after the other.

3. picard crashing: i am pretty sure picard ships with a 32bit python on windows so ~2gb is just the "natural" memory limit for a 32bit process. nothing the plugin can do about that, but maybe try to use less memory.

4. to be done: probabably. easy: unlikely.

caching to disk is something i wanted for a while. also with persistence from session to session. that should solve the crash if done right. python is weird sometimes when it comes to actually freeing up memory, but should be doable. similar issue as with next point: it adds dependencies i have to include and more code i have to maintain.

another problem-point is the picard task queue the plugin integrates with. it is kind of a moving target: picard internal, not documented api. but moving to a complete standalone implementation increases plugin size and introduces dependencies, that i'd have to ship for mac/linux/win support. picard does not even include all of the python standard library in their windows build (eg configparser).

5. so for now i'd just recommend to tag in smaller batches.

i am marking this as 'wontfix' as i don't see me having time for this, but if you want to work on memory optimizations, feel free to send pull requests and i'll try to find time to review and merge.

[pumalife]

I want to retag that many tracks because they are all clean (tagged, organized and high probability of being recognized as perfect by picard) albums that have either no last.fm tag data, or used the last.fm plus plugin. so different rules, different amount of values, different tag names. Just want to update them all. Now using your plugin with uniquely configured search lists. Thanks for breaking it down. 32bit of course.

[fdemmer]

just adding an idea here: write all tag info from last.fm to sqlite and do more via sql than on in-memory data.

[Enverex]

Was any progress ever made on this? I'm running into a wall with Picard using 2GB of RAM at which point it just seems to... stop. Shouldn't be a RAM limitation as it's 64bit Python on Linux.

Running Picard in debug mode shows these lines prior to it just... freezing.

I: 09:07:19 Last.fm.ng - occasion: chillout (196)
I: 09:07:19 Last.fm.ng - occasion: metatag: comment:Songs-DB_Occasion
I: 09:07:19 Last.fm.ng - comment:Songs-DB_Occasion = Chillout
I: 09:07:19 Last.fm.ng - category: 4 tag(s) before threshold filter:
I: 09:07:19 Last.fm.ng - category: singer-songwriter (42), handclaps (40), amazing (40), male vocalist (4)
I: 09:07:19 Last.fm.ng - category: score threshold: 21.0 (50%)
I: 09:07:19 Last.fm.ng - category: 3 tag(s) filtered:
I: 09:07:19 Last.fm.ng - category: singer-songwriter (42), handclaps (40), amazing (40)
I: 09:07:19 Last.fm.ng - category: metatag: comment:Songs-DB_Custom2
I: 09:07:19 Last.fm.ng - comment:Songs-DB_Custom2 = Singer-Songwriter; Handclaps; Amazing
I: 09:07:19 Last.fm.ng - country: 3 tag(s) before threshold filter:
I: 09:07:19 Last.fm.ng - country: british (582), uk (8), england (6)
I: 09:07:19 Last.fm.ng - country: score threshold: 407.4 (70%)
I: 09:07:19 Last.fm.ng - country: 1 tag(s) filtered:
I: 09:07:19 Last.fm.ng - country: british (582)
I: 09:07:19 Last.fm.ng - country: metatag: comment:Songs-DB_Custom4
I: 09:07:19 Last.fm.ng - comment:Songs-DB_Custom4 = British
I: 09:07:19 Last.fm.ng - city: 1 tag(s) before threshold filter:
(nothing since then, nearly an hour ago)

It's been sat at 100% CPU usage since then, ~2GB RAM usage and the interface is completely unresponsive.

I'm trying to tag all the albums for artists who's names start with A which is roughly 1200 tracks (trying to break this up any smaller becomes impossible to keep track of).

[fdemmer]

32-bit python cannot address more than 2GB memory.

no work has been done/planned to improve memory usage of the plugin.

pull-requests are welcome.

On 2017-09-15 10:38, Benjamin Hodgetts wrote:

Was any progress ever made on this? I'm running into a wall with Picard using 2GB of RAM at which point it just seems to... stop. Shouldn't be a RAM limitation as it's 64bit Python on Linux.

Running Picard in debug mode shows these lines prior to it just... freezing.

|I: 09:07:19 Last.fm.ng - occasion: chillout (196) I: 09:07:19 Last.fm.ng

• occasion: metatag: comment:Songs-DB_Occasion I: 09:07:19 Last.fm.ng - comment:Songs-DB_Occasion = Chillout I: 09:07:19 Last.fm.ng - category: 4 tag(s) before threshold filter: I: 09:07:19 Last.fm.ng - category: singer-songwriter (42), handclaps (40), amazing (40), male vocalist (4) I: 09:07:19 Last.fm.ng - category: score threshold: 21.0 (50%) I: 09:07:19 Last.fm.ng - category: 3 tag(s) filtered: I: 09:07:19 Last.fm.ng - category: singer-songwriter (42), handclaps (40), amazing (40) I: 09:07:19 Last.fm.ng - category: metatag: comment:Songs-DB_Custom2 I: 09:07:19 Last.fm.ng - comment:Songs-DB_Custom2 = Singer-Songwriter; Handclaps; Amazing I: 09:07:19 Last.fm.ng - country: 3 tag(s) before threshold filter: I: 09:07:19 Last.fm.ng - country: british (582), uk (8), england (6) I: 09:07:19 Last.fm.ng - country: score threshold: 407.4 (70%) I: 09:07:19 Last.fm.ng - country: 1 tag(s) filtered: I: 09:07:19 Last.fm.ng - country: british (582) I: 09:07:19 Last.fm.ng - country: metatag: comment:Songs-DB_Custom4 I: 09:07:19 Last.fm.ng - comment:Songs-DB_Custom4 = British I: 09:07:19 Last.fm.ng - city: 1 tag(s) before threshold filter: (nothing since then, nearly an hour ago) |

It's been sat at 100% CPU usage since then, ~2GB RAM usage and the interface is completely unresponsive.

I'm trying to tag all the albums for artists who's names start with A which is roughly 1200 tracks (trying to break this up any smaller becomes impossible to keep track of).

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/fdemmer/Picard-Last.fm.ng-Plugin/issues/20#issuecomment-329719176, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmgv96CrLJ5tup8G_jUtgoZmq3dN-Neks5sijdpgaJpZM4GzYYo.

-- Florian Demmer Kundratstrasse 6/12/13 1100 Vienna, Austria

mail: [email protected]

[Enverex]

I did say... "Shouldn't be a RAM limitation as it's 64bit Python on Linux".

Side note, this bug report appears to have disappeared, I've had to reply by email.

[fdemmer]

i don't see it either in the tracker, don't recall deleting it :(

still, pull requests are welcome.

On 2017-09-15 12:26, Benjamin Hodgetts wrote:

I did say... "Shouldn't be a RAM limitation as it's 64bit Python on Linux".

Side note, this bug report appears to have disappeared, I've had to reply by email.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/fdemmer/Picard-Last.fm.ng-Plugin/issues/20#issuecomment-329744251, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmgvxUYAgC0pT9kAKAV_Su0zt6soD9Vks5silDjgaJpZM4GzYYo.

[time2kite]

Picard 1.4.2, Windows 10, latest lastfmng Plugin Getting lastfmng to work is a matter of chance. Most of the time, Picard just sits there with "loading album information", no matter if I have 10 or 3000 tracks. Memory usage is constantly rising until it crashes after maybe 15 minutes or so at ~ 1.5 GB. Seems to be a serious memory leakage. Restarted PC but no effect.

Only once I managed to tag 500 files at once. Most of the time it fails with 30 or less songs, so load does not seem to be the problem. Strangely, once it starts to recognize albums (if ever), it seems to finish with all tracks. If I deactivate lastfmng, I can easily tag 2000 songs at a time.