Auxio
Auxio copied to clipboard
OxygenCobalt
Music caching
Even though #128 is complete, it is still not enabled by default and is actually labeled experimental. This is because loading with #128 tends to be quite slow, and thus not fully ready to become Auxio's default and unchangeable behavior. This PR is about implementing a cache strategy to reduce the time spent on extracting metadata wherever possible.
The general goal is for me to:
- Quickly start the app without having to reload the music library
- Unless "Automatic Reloading" is enabled, which in that case I will have to reload every time since any files could have changed while the app was dead.
- Ignore files whose metadata didn't change significantly. This is mostly comparing the last-modified times between the cached music and the music that is currently being loaded.
I could take this in two ways, either I go with a Full Cache, or a Mini Cache.
- A Full Cache would consist of the entire music library (Songs, Albums, Artists, Genres) all being stored in a relational database. This solves 1 and 2, but also has the added bonus of using less memory, something I'm slightly concerned when it comes to large libraries. The issue with this is that any music access now becomes IO work, which I feel like would ruin the playback experience due to the delay incurred by reading from a database.
- A Mini Cache on the other hand would just be a dump of song metadata and their timestamps. When normally loading, the music library is constructed from the cache, and when rescanning music, the timestamps are compared to see if a song file meaningfully changed. This is generally the more technically feasible solution and the one I prefer.
Doing this will finally allow #128 to graduate from being experimental, which would thus unblock a few other issues related to it.
mini cache ftw! i actually do something similar in my music tagging app, where on first scan i dump all metadata that is read off of music files into a key-value pair db (key: filename, value: metadata) and then all other app starts are really quick! i just check if all songs have the metadata in the db and if not, i add the ones that don't. if a song changes it's metadata, you could just have a manual refresh/flush db button in settings or do this every-so-often while the app is in bg
That's a bit of a different approach.
I've been thinking about this futher, and:
- I don't think I'll use the file name as much as I'll use the MediaStore ID. This is okay, as if a file is modified, it's ID will be invalidated, and if a new file takes up the ID of an invalidated file, the last modified time will differ and thus the cache instance will be invalidated.
- I'll likely cache structured data representing the tags I do leverage, and not a dump of the metadata. This allows me to save space for libraries that have an absurd amount of tags.
- Auxio likely won't load solely from the cache. I'll take the performance hit in order to prevent ID drift.
- Reload music might be split into "Reload music" and "Rescan music" to differentiate between reloading with the cache and reloading without the cache.
okay 👌 . just make sure the difference between reload and re-scan is explained under the buttons, as it can be confusing from first glance what means what.
the timestamps are compared to see if a song file meaningfully changed.
Not sure I agree with this in particular. What if say I've changed some Id3 tags, or added/removed cover art from a music file, would those changes not be "meaningful" enough to actually be updated in the app?
Not sure I agree with this in particular. What if say I've changed some Id3 tags, or added/removed cover art from a music file, would those changes not be "meaningful" enough to actually be updated in the app?
Filesystems generally keep track of when a file is last modified, that's what the cache is making use of, does your ID3 tagger somehow not trigger that?
If it doesn't, then there's still the option for reloading music without doing the fast path as mentioned two comments above
- Reload music might be split into "Reload music" and "Rescan music" to differentiate between reloading with the cache and reloading without the cache.
Yeah, @intrnl is right. Auxio will likely check for something like this in order to determine whether a file hasn't meaningfully changed:
if (new id == cached id && new date added == cached date added && new date modified == other date modified) {
// This file has not changed
}
No tag comparison is done, as I'm going to assume that OEMs aren't crippling the date modified and added fields.
Filesystems generally keep track of when a file is last modified, that's what the cache is making use of
Right, I see. I think I misunderstood how the feature would work.
Done. This should arrive in 3.0.0. Here's a demo below that shows how insane the speedup is now, even with the "Ignore MediaStore tags" music loader.
https://user-images.githubusercontent.com/65370175/192182855-d9d4531f-9fb1-4d78-9443-2f84493bf820.mp4
here's from my current library (9157 songs) now that v3.0.0 is out
v2.6.4: 1m10s v3.0.0: 1m20s (cold) v3.0.0: 13s (cache)