chainlink-solana icon indicating copy to clipboard operation
chainlink-solana copied to clipboard

Log pruning for Solana LogPoller

Open reductionista opened this issue 9 months ago • 1 comments

Description

This adds a PruneLogs method to lp.filters which gets run by the background worker thread every 10 minutes, just after PruneFilters is run. A 9 minute deadline on the context is set, to make sure a previous pruning run never interferes with the next one.

The filters are shuffled randomly and iterated over. This could potentially help in a case where the db is overloaded enough that not all filters can be pruned, and some have a lot more logs than others. It avoids a situation where the first few filters have too many logs to successfully prune at once, in which case it might never make an attempt to prune the rest of the filters.

orm.PruneLogsForFilter prunes any logs which have expired (based on their expiresAt column, which is already set during log intake based on the Retention field of the filter). And in the same query, it also removes older logs in excess of the most recent MaxLogsKept logs, as specified by the filter. Each filter requires a single query, whose timeout is set to 1 minute.

As long as everything is functioning normally, these DELETE queries should be fast (< 100ms), not needing anywhere near that much time. Most of the above are just preventive measures to help recover from a hypothetical spike in log volume, where the db is already under high load and then millions of logs somehow get inserted before the pruner can run.

NONEVM-1261

Required for CI to pass until https://github.com/smartcontractkit/chainlink/pull/16915 gets merged (but okay to merge in either order):

core ref: 443ded9c1f5f051e63591fc075bbdcce3280f5d4

reductionista avatar Mar 26 '25 02:03 reductionista

Quality Gate failed Quality Gate failed

Failed conditions
46.5% Coverage on New Code (required ≥ 75%)
6.8% Duplication on New Code (required ≤ 5%)

See analysis details on SonarQube