Evict wave usage metrics after 120 days

[munishchouhan]

This PR will do the following:

1. add functionality to scan and delete keys with specific dates.
2. add corn to run every day and delete keys with current date - 120

[pditommaso]

I think this can be implemented more easily by specifying the value ttl in the redis map

https://github.com/seqeralabs/wave/blob/b2790ee8bc60e1338fb73d6825770f3168dbdc1c/src/main/groovy/io/seqera/wave/service/counter/AbstractCounterStore.groovy#L42-L42

See this pattern

https://github.com/seqeralabs/wave/blob/b2790ee8bc60e1338fb73d6825770f3168dbdc1c/src/main/groovy/io/seqera/wave/service/cache/AbstractCacheStore.groovy#L70-L82

[munishchouhan]

The problem is we are using hash in this case and TTL is normally applied on the key and not on the fields, I will dig further to see if there is any way to achieve this for a filed in a key

[munishchouhan]

here is a thread saying its not possible https://stackoverflow.com/questions/16545321/how-to-expire-the-hset-child-key-in-redis

[munishchouhan]

There is this PR that implemented expiration on hashes https://github.com/redis/redis/pull/13172 its available on version 7.2

[pditommaso]

What's preventing expiring the key ?

https://github.com/seqeralabs/wave/blob/b2790ee8bc60e1338fb73d6825770f3168dbdc1c/src/main/groovy/io/seqera/wave/service/metric/impl/MetricsServiceImpl.groovy#L106-L107

[munishchouhan]

This is the key in our case https://github.com/seqeralabs/wave/blob/909bda14000957e380e4e0414948693d122ee3e8/src/main/groovy/io/seqera/wave/service/metric/MetricsCounterStore.groovy#L39-L41

This is field: https://github.com/seqeralabs/wave/blob/b2790ee8bc60e1338fb73d6825770f3168dbdc1c/src/main/groovy/io/seqera/wave/service/metric/impl/MetricsServiceImpl.groovy#L106-L107

see here: https://github.com/seqeralabs/wave/blob/909bda14000957e380e4e0414948693d122ee3e8/src/main/groovy/io/seqera/wave/service/counter/AbstractCounterStore.groovy#L42

[pditommaso]

I think solution 1 here, would make the trick

[munishchouhan]

I think solution 1 here, would make the trick

I caanot access this

[pditommaso]

TLDR;

Set the TTL on the entire hash map key:

HINCRBY myhash field 1
EXPIRE myhash 60  # Set TTL of 60 seconds on the entire hash map

Essentially an inc followed by an expire operation

[pditommaso]

Actually this should be done only for *day* metrics, therefore the ones having /d/<date> in the key

[munishchouhan]

There is this method for expiry. but it still only required key and not field

[pditommaso]

What's wrong with that?

[munishchouhan]

What's wrong with that?

ok you are correct, if i use expiry it does work I will develop it using this

[munishchouhan]

@pditommaso I have added two test failing and success to show why expire with key wont work, because we need to evict the field in a key and not the whole key, i am using 1 second as ttl

failing test: https://github.com/seqeralabs/wave/blob/abd522a5b9caa26cc22f7405bd7b36f45c7567e6/src/test/groovy/io/seqera/wave/service/counter/impl/RedisCounterProviderTest.groovy#L87

success test: where everything will be null https://github.com/seqeralabs/wave/blob/abd522a5b9caa26cc22f7405bd7b36f45c7567e6/src/test/groovy/io/seqera/wave/service/counter/impl/RedisCounterProviderTest.groovy#L100

[munishchouhan]

the problem is we are using redis hash, where we have to fields, key and field expirying key does not help our use case

[pditommaso]

Considering this is not critical. I'd say to park until Redis 7.2 is available, so that we can use expired on the field

[munishchouhan]

Hash expiration functionality will be available in jedis 5.2.0 https://github.com/redis/jedis/releases/tag/v5.2.0-beta4 and required redis 7.4

[munishchouhan]

Hash expiration functionality has been added, but the jedis dependency is still in beta, once the final version is released. I will make this pr ready for review

[munishchouhan]

Redis v 5.2.0 has been released https://github.com/redis/jedis/releases/tag/v5.2.0

Now this PR needs Redis 7.4 or later