[IDEA] buffer filter should set an expiration on redis lists

[frikilax]

What is your feature about ? the buffer filter currently has a "flaw" in data management: if buffer sources are modified after some time, the old sources (aka redis lists) will probably not be empty, this create a "memory leak" of sorts, as those lists won't probably ever be deleted from Redis, stacking up and using unnecessary memory

If your request is about an existing filter/functionality This Feature Request is about the buffer filter

Describe what you'd like To prevent that from happening, a minimal change needs to be done: when the buffer starts or triggers to get all cached data from Redis, it should set/reset an expiry on the key

Thanks to that, 2 problems are solved:

• if sources names change inside the buffer filter, old sources will eventually be cleared out of Redis -> no "memory leak"
• if Darwin happens to have some downtime, this technique can remove obsoleted data from Redis

Additional context The expiry value should be coherent with data longevity AND the frequency at which the buffer needs to fetch cached data, a sensible option could be to take the maximum value between twice the fetch interval, and a fixed value corresponding to data "freshness" (time before considering data is obsolete). In all cases, the expiry value should be strictly more than the configured fetch interval, to avoid losing data before fetching it