Feature Request: Define custom actions when cache become invalidated

[lcdsmao]

Is it possible to execute some custom actions when memory cache become invalidated?

I am trying to integrate Jetpack Paging with Store. The DataSource required us to call invalidate when we want to stop paging.

Currently, I am using a map to cache the latest result and invalidate old one from the fetcher:

val dataSourceMap = mutableMap<Key, WeakReference<PagedList>>()
val store = StoreBuilder.fromNonFlow { key: Key ->
  val pagedList = createPagedList(key)
  dataSourceMap[key]?.get()?.dataSource?.invalidate()
  dataSourceMap[key] = WeakReferene(pagedList)
  pagedList
}.build()

Maybe it will be convenient if StoreBuilder can provide a onCacheInvalidated method:

val store = StoreBuilder.fromNonFlow { key: Key ->
  createPagedList(key)
}.onCacheInvalidate { pagedList -> 
  pagedList.dataSource.invalidate() 
}.build()

Edit: Use WeakReference

[ychescale9]

Apart from the cache being invalidated, there are other reasons why the fetcher is triggered (e.g. StoreRequest.fresh(key)).

Looks like you just want to invalidate the data source of existing pagedList before a new pagedList loaded from the fetcher replaces the old generation?

Would something like this work?

pagedListStore.stream(StoreRequest.cached(key, refresh = true)) // could be any type of request
    .onEach { response ->
        // invalidate existing paged list data source (if any) before replacing it with new one loaded from fetcher
        if (response is StoreResponse.Loading && response.origin == ResponseOrigin.Fetcher) {
            response.dataOrNull()?.dataSource.invalidate()
        }
        
        // process pagedList...
    }
    .launchIn(uiScope)

And since Store supports multicasting you could also generate a separate Flow<PagedList> from the store for performing this kind of maintenance work, to keep the processing of the main data stream clean.

[lcdsmao]

Apart from the cache being invalidated, there are other reasons why the fetcher is triggered (e.g. StoreRequest.fresh(key)).

I am wondering the cache system is like a collection of key-value, so trigger the fetcher successfully via the same key (e.g. StoreRequest.fresh(key)) will result in the old value being invalidated. Is my understanding of the cache system wrong? 😢

Looks like you just want to invalidate the data source of existing pagedList before a new pagedList loaded from the fetcher replaces the old generation?

Sorry for my unclear description. I want to invalidate the data source of the existing pagedList after a new pagedList loaded successfully from the fetcher. If my understanding of the cache system is correct, the existing pagedList will be invalidated, right?

---

I have two questions about the sample code:

pagedListStore.stream(StoreRequest.cached(key, refresh = true)) // could be any type of request
    .onEach { response ->
        // invalidate existing paged list data source (if any) before replacing it with new one loaded from fetcher
        if (response is StoreResponse.Loading && response.origin == ResponseOrigin.Fetcher) {
            response.dataOrNull()?.dataSource.invalidate()
        }
        
        // process pagedList...
    }
    .launchIn(uiScope)

1. It seems like if response is StoreResponse.Loading, then response.dataOrNull will always give null. Maybe the operator scanReduce can do a better job?
2. How to separate this invalidation logic and let it execute lazily? Since if there is no cache available, then the store will execute the fetcher eagerly. Currently, I mainly used the get and fresh method and my code look like this:

class MyViewModel : ViewModel() {

    private val store: Store<Key, PagedList<Value>> = ...

    init {
        // Separate the invalidation logic, the fetcher will be executed if there is no cache.
        // However, the excepted behavior is to lazily execute the fetcher after the user signed in, 
        // which means after the first execution of [store.get] or [store.fresh].
        viewModelScope.launch {
            store.stream(StoreRequest.cached(key, refresh = false))
                .invalidationLogic()
                .collect()
        }
    }

    // Call this after onCreate of activity.
    suspend fun get() = if (isSignedIn) store.get(key) else null

    // Call this when doing pull to refresh
    suspend fun fresh() = if (isSignedIn) store.fresh(key) else null
}

Any suggestions will be appreciated! 🙇‍♂️

[ychescale9]

Oh now I understand your requirements 😃

I am wondering the cache system is like a collection of key-value, so trigger the fetcher successfully via the same key (e.g. StoreRequest.fresh(key)) will result in the old value being invalidated. Is my understanding of the cache system wrong? 😢

No your understanding is correct. When I first saw "invalidated" I was thinking about evictions or expirations in the in-memory cache, whereas "invalidation" here means new values replacing existing ones.

I think what you really want is a notification whenever a new entry loaded from the fetcher is about to replace existing entries (if any) in the in-memory cache and/or the persister (source of truth). In that case your dataSourceMap solution is probably the closest to what you want without introducing new APIs.

1. It seems like if response is StoreResponse.Loading, then response.dataOrNull will always give null. Maybe the operator scanReduce can do a better job?

Hmm I would expect the previously cached data to be present even when the current state is StoreResponse.Loading, but I haven't tried this or looked at the code yet.

2. How to separate this invalidation logic and let it execute lazily? Since if there is no cache available, then the store will execute the fetcher eagerly. Currently, I mainly used the get and fresh method and my code look like this:

I think you can also conditionally collect the Flow based on isSignedIn? Although I'd probably try to model it such that the ViewModel has a single stream of inputs (Actions) and a single stream of outputs (States).

However, I agree that it makes more sense to be able to do things like invalidating stale data source when configuring the store.

Do you have a minimum sample app with store and paging integrations that we can play with? It would make it a lot easier to explore different use cases and get a feel of how the new APIs should look like if we want to better support this. 😃

[lcdsmao]

@ychescale9

I created a playground repo about my requirement that using Store to cache PagedList. https://github.com/lcdsmao/PagingMemoryCacheStorePlayground

Can you have a look at it, please? 😄

[ychescale9]

Thanks. Will take a look.