Use configurable or shorter TTL when caching previews in Redis

[jnm]

Is your feature request related to a problem? Please describe. Enketo caches all XSLT transformations for 30 days: https://github.com/enketo/enketo-express/blob/dde231341062b5660563efe2b60a84938b2e7233/app/models/cache-model.js#L9 In a large deployment, it's easy to accumulate many gigabytes of cache in Redis for previews. Redis requires all data to fit into RAM, and I'd argue that it's not a good use of system resources to give previews, which by definition are not used for large-scale data collection, equal footing with fully-fledged surveys.

Describe the solution you'd like I'd like to be able to configure the Redis TTL / expiry / expiration time independently for both data-collection surveys and previews, e.g. persist surveys for 21 days and previews for 1 day. If that's too ambitious, then configuring the TTL for previews alone would be helpful. Failing that, even hard-coding a shorter expiry for previews would be an improvement.

Describe alternatives you've considered

• Manually purging cached previews from Redis: unsustainable, but possible with servers that have distinguishing OpenRosa URLs for previews vs. real surveys;
• Setting maxmemory and maxmemory_policy in the Redis configuration to purge cache entries when memory consumption gets too high: okay, but then previews (often used one time only) crowd out forms with real traffic;
• Changing the hard-coded behavior in a fork :upside_down_face:

Additional context KoBo is willing to contribute a PR if a maintainer indicates that a change like this would be acceptable.

[MartijnR]

Just to be sure, you're not referring to previews with an form query parameter, right? (Because those are not cached I believe: https://github.com/enketo/enketo-express/blob/master/app/controllers/transformation-controller.js#L53)

I remember KoBo used those query parameter views in the past but maybe you've switched because of the limitations with media and external data in those views (OC has also switched). If so, are you using some kind of temporary server_url (one per KoBoToolbox installation?) to "fake-launch" these surveys perhaps?

[jnm]

Hi @MartijnR, right! We switched to using api/v2/survey/preview/iframe. We do pass along a temporary server_url (and a temporary form_id) in the POST data, so I can clean out our Redis cache by purging keys like ca:kobo.bigagency.info/api/v2/asset_snapshots*.

What I was hoping, though, was that we could identify surveys by virtue of them being created with the api/v2/survey/preview endpoint, and then treat their caching differently.

Honestly, I just now noticed this: https://apidocs.enketo.org/v2#/delete-survey-cache. I suppose if it's not easy to distinguish between previews and fully-fledged surveys on the Enketo side, then the requesting service (like KoBo) could ask Enketo to clear previous cached transformations of the form.

[jnm]

I didn't get the solution I wanted with https://apidocs.enketo.org/v2#/delete-survey-cache, but in the absence of any evidence to the contrary, let's assume that my use of it is simply flawed.

What does seem to work is setting maxmemory on the Redis cache server and specifying an eviction policy. I'm using maxmemory-policy volatile-lfu.