[DNM] sql: add an option to cache query results

[ivan4th]

With {"main":{db-query-cache": true}} option (false by default), some of the query results are stored in memory.

Motivation

On the nodes with high peer counts, a lot of CPU time is spent on handling SQLite queries and also GC resulting from multiple database requests and same data being serialized over and over again. With the query cache enabled on such nodes, the memory use doesn't increase much, as normally there's large number of requests "in flight" most of the time, and cached data are reused to handle these requests w/o further temporary allocations.

Changes

Currently, ATX ID lists per epoch, ATX blobs and ActiveSet blobs are cached. Also, serialized ATX ID lists are cached by the fetcher. This reduces CPU load and GC pressure on the nodes that serve a lot of fetch requests.

Test Plan

Verified on a node with 8K+ peers

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 78.09917% with 53 lines in your changes are missing coverage. Please review.

Project coverage is 79.8%. Comparing base (9132687) to head (82482bf).

Files	Patch %	Lines
fetch/handler.go	32.7%	35 Missing and 2 partials :warning:
sql/atxs/atxs.go	80.6%	3 Missing and 3 partials :warning:
sql/querycache.go	94.1%	3 Missing and 3 partials :warning:
sql/activesets/activesets.go	88.8%	1 Missing and 1 partial :warning:
sql/database.go	90.4%	1 Missing and 1 partial :warning:

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #5465    +/-   ##
========================================
  Coverage     79.8%   79.8%            
========================================
  Files          270     271     +1     
  Lines        27186   27347   +161     
========================================
+ Hits         21713   21843   +130     
- Misses        3946    3971    +25     
- Partials      1527    1533     +6     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[poszu]

@ivan4th I can't figure this out - what's preventing this cache from growing without bounds? When are "old" entries removed?

[ivan4th]

@poszu the cache is not purged at the moment. The option is only intended for nodes with a lot of RAM and which also have to serve an (unusually) large number of epoch info / hash requests, and not the usual end-user machines. It is assumed that RAM is large enough to hold everything that's in the db, so to speak. Later, the query cache can be made LRU (and maybe combined with the existing LRU cache, somehow) to make it usable on the normal user nodes, and also to keep working when the database becomes too large.

[ivan4th]

Adding DNM label b/c the change is not strictly required for now, and should be considered an "extra safeguard" measure

[ivan4th]

bors try

[spacemesh-bors[bot]]

try

Build succeeded:

• ci-status
• systest-status

[ivan4th]

bors try

[spacemesh-bors[bot]]

try

Build succeeded:

• ci-status
• systest-status

[ivan4th]

Updated the PR with LRU support so that the caches don't grow indefinitely. For now, the caching is only done for ATX blobs, epoch ATX IDs and ActiveSet blobs; more objects will be included after both this PR and #5562 are merged

[ivan4th]

bors merge

[spacemesh-bors[bot]]

Pull request successfully merged into develop.

Build succeeded:

• ci-status
• systest-status