Cache
Cache copied to clipboard
hyperoslo
Get cache all objects
I need this feature as well for my app so I'm about to start poking around at a potential implementation.
@3lvis So, I've only dug into the code a bit, but I have a few questions.
It looks like all data objects are just stored in the same directory? I'm guessing that the primary key is hashed and the file name is that hash? If yes, is there a way to iterate over the objects and know the type?
(Assuming disk storage, memory can come second):
There would need to be a way to track which objects belong to a type. It could be that all objects of a type are stored within a specific directory, thus knowing that if the data object is the directory that it is a certain type. Then the directories would have a mapping to the type. Or, an array of primary keys could be stored for each type and then on the get all objects for type call, it gets that primary key array and loops through trying to get each objects with the existing entry function. Compact map the result and return all those objects.
I think that in large scale instances, this could be inefficient. But I think that would be a simple solution.
Loosely related, I spoke with my roommate for a while about the concept of getting all of an object. His point was to look at all the cases of using a get all object of type function and make sure that it is actually needed. In my application, there are 33 instances of getting all the objects. For those a third were to get one object of an unknown key (so store the key) and the others were actually backed by an API call and needed an API call every time. Thus, in my application, I can remove the get all objects function for things that are more correct or performant.
I would use this functionality as well for the project I'm working on.
I believe that I can work around this for now using direct file manager access but it's not ideal since the keys are opaque.
Thanks for the follow up everyone! As far as I know the hardest part is to get a sense of the keys, if we had the keys we could be able to iterate over them and return all the objects.
What's the purpose of adding something like this? In which scenarios you need to return all cache objects?
Thanks for your patience, just trying to understand your use cases.
In our case, the reason for needing to get all the keys is to update the cache when the set of data objects has changed. We have a set of data [a,b,c,...] that is sourced by an API server. This set of data can change for whatever reason to [a,b,d]. The client needs to walk through the data in the cache and delete c since it is no longer valid. A time-based aging approach won't work for us because since we want the data to stick around indefinitely. We also don't get the full set of all data with each call as we only return changes to the full set, so wiping the entire cache and repopulating also won't work. We also can't simply leave the data in the cache because not only is that a waste of space but more importantly there are PII concerns with keeping old data on the device.
The envisioned use case is to find the differences between the server set and the cache set and remove old items in the cache as they expire.
Ahhh, got it. Thanks @richardpineo! If anyone wants to have a got at this I'll be happy to review and merge.
I did it for memory storage. I would have liked to do it for DiskStorage but as the filename is created using a hash it is not possible. However, it is a good first approximation.
https://github.com/hyperoslo/Cache/pull/280
