Caching and Pagination ?

[danramteke]

Hi, I'm having a little trouble getting Rails caching to play nicely with this gem. Here's some psuedo code that matches my code fairly well:

# Shows HTTP paging headers, but loads everything into memory
def user_posts
  cache_key = {
    user_id: current_user.id, 
    latest_post: current_user.posts.maximum(:updated_at)
  }

  posts = Rails.cache.fetch(cache_key) do
     current_user.posts.order('created_at DESC')
  end
  render json: paginate(posts), each_serializer: PostSerializer
end

If I do this, I will get the api_pagination HTTP headers. However, I'm loading all of the current_user's posts into memory, which seems like a bad thing.

If I paginate inside the caching block (like below), then I don't get any api_pagination HTTP headers.

# Doesn't show HTTP paging headers, but doesn't load everything into memory
def user_posts
  cache_key = {
    user_id: current_user.id, 
    latest_post: current_user.posts.maximum(:updated_at), 
    page: params[:page]
  }

  posts = Rails.cache.fetch(cache_key) do
     paginate(current_user.posts.order('created_at DESC'))
  end
  render json: posts, each_serializer: PostSerializer
end

Can someone point me in the right direction? I'm having a hard time googling for API Pagination and Rails caching. Perhaps I'm missing something fairly obvious. I'm using kaminari, if that helps.

Thanks so much!

[davidcelis]

@danramteke Is your app open source, by any chance? If not, would you be able to reproduce this in a fresh rails app using kaminari and api-pagination and send me the app's code?

[davideghz]

Just had the same problem. paginate overrides Rails' render method.

This override is skipped if the application serves the cached results, so in order to get the correct (full) headers in case the request hits the cache, we have better to paginate what the cache returns instead of paginating inside the cache block.

Something like the following should work:

...
posts = Rails.cache.fetch(cache_key) do
  current_user.posts.order('created_at DESC')
end
paginate json: posts, each_serializer: PostSerializer
...

[jacek213]

@davideghz as the author stated, this way you're going to put all posts into cache, and u might have milions of them, so that's not a good solution.

[alemata]

Hi!! Did anyone find a solution for this? I am having the same issue...

[davidcelis]

Hey all, sorry that there's still an issue with this. I'm still happy to look into this, but I haven't experienced the issue, so having access to the source code of one of these apps that I can run locally would help. Or even just a dummy Rails app that's set up to reproduce the problem.

[alemata]

Hello @davidcelis! Thanks for the reply... I have created a toy project to test this (https://github.com/alemata/pagination_and_cache)

I know this is not the ideal code to test but it's an idea of the issue

Ideeally we should be able to paginate... store that page result in cache and send the responde with pagination headers...

Sorry if I am being not so clear... please ping me with any question

[alemata]

@davidcelis were you able to review this?! was the project helpfull?

[alemata]

Ok I think I make it work...

# call paginate outside cache to add headers
paginated = paginate Property.all
paginated_collection = Rails.cache.fetch(Property.cache_key(properties)) do
  # Store just the collection in cache (`paginate method returns the collection`)
  paginated.to_a
end

paginated_collection

I am not sure if there is a nice way to improve this inside the gem...