stashapp
[RFC] SSD Cache Management
Scope
Pre-load subset of library onto SSD for better performance
Long Form
I am planning on implementing this for my use case over the next few weeks but am willing to generalize this feature and make it OSS if others are interested.
Motivation
I like to skip around a lot when I watch videos and sometimes have to wait for it to buffer for a few seconds even on local network with Ethernet. I will add an m.2 SSD 800gb to my Synology where I will store a copy of a selection of videos from my library.
Secondary goal: Duplicate the cache onto a portable drive before traveling to places with bad internet
Functionality
I will describe how I use my library and how we can use the pattern of behavior to intelligently populate the cache. I have around 20 saved filters that I use. The videos are sorted randomly. I will watch a few and then go to a different filter etc. Most filters have the condition "play_count < 1" although some do not. If I run out of videos for a filter I will increment the count. But each filter has 100-1000 videos so I won't need to do that for a long time.
How I will populate the cache is to decide a number n_i for each filter_i that is the number of videos to load into the cache. I will add a tag to those videos to identify them as cached and then copy the files into the cache. Every night when I sleep a cron job will run that will find cached videos that have been watched, delete them from the cache (and remove the tag), and add new videos so that there are n_i unwatched videos for each filter_i.
Identified areas of risk
- few videos are unwatched for a filter and need to increment playcount condition
- Filters that don't have playcount condition. How do we determine that they have been watched and purge them from the cache? Maybe use most recently watched date to determine?
- Usage session during cache refresh.
- Wide range of file sizes for a single filter.
- Maintain parity between filters in ui and filters in cache system. If a named filter is modified, how do we ensure that the cache will be populated with the updated filter? Should the change apply to existing videos in the cache, or just videos entering the cache
Examples
Can provide some pseudo code if my intentions are unclear
Conclusion
Please discuss if this would be useful to OSS and build properly. Otherwise I will just implement with a python script. I could upload that to community plugin repo if there is interest in that
The implementation sounds like it would only get out of hand and doesn't exactly align with the core ethos of stash which is local content management. This would also have to expand it's own scope outside of stash itself because of the remote access, which would probably be better suited for a plugin. Notably
- management of an external library, with known duplicates
- validation and processing of that external library
- browser client must have knowledge of the external library and ability to access the remote library
I can see why the secondary goal may be useful, but it's probably something something to do with a plugin or companion app.
I might suggest if you're having trouble with videos over the network the issue may be your drive has spun down or your server is slow, rather than an SSD improving it, fixing those other items may help it more. Unless viewing enormous numbers of small files (eg large image galleries) hard drives are actually more than fast enough. I don't have this 'buffer for a few seconds' problem unless my CPU is busy doing something else, or the drive has gone to sleep.
