refactor(cache): Add database interface for blob paths + refactor BoltDB to use interface

[chofnar]

What type of PR is this?

enhancement

Which issue does this PR fix: Part of #564

What does this PR do / Why do we need it: This PR adds a database interface that allows the usage of vendor-specific database services by implementing their drivers. The usage of BoltDB was refactored to use this new interface.

Other comments: This PR is a result of the split of #588, which now depends on this one. The dynamodb integration can be found here

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov[bot]]

Codecov Report

Merging #667 (7144db0) into main (11ec261) will increase coverage by 0.08%. The diff coverage is 67.74%.

:exclamation: Current head 7144db0 differs from pull request most recent head 7ad8818. Consider uploading reports for the commit 7ad8818 to get more accurate results

@@            Coverage Diff             @@
##             main     #667      +/-   ##
==========================================
+ Coverage   88.71%   88.80%   +0.08%     
==========================================
  Files          73       74       +1     
  Lines       13847    13912      +65     
==========================================
+ Hits        12285    12354      +69     
+ Misses       1216     1214       -2     
+ Partials      346      344       -2     

Impacted Files	Coverage Δ
pkg/storage/cache/boltdb.go	56.30% <56.30%> (ø)
pkg/storage/cache.go	60.00% <62.50%> (+6.41%)	:arrow_up:
pkg/api/config/config.go	89.55% <100.00%> (+0.32%)	:arrow_up:
pkg/api/controller.go	93.35% <100.00%> (+0.38%)	:arrow_up:
pkg/cli/root.go	95.70% <100.00%> (+0.28%)	:arrow_up:
pkg/extensions/sync/utils.go	93.11% <100.00%> (ø)
pkg/storage/common.go	97.26% <100.00%> (ø)
pkg/storage/local/local.go	82.15% <100.00%> (+0.66%)	:arrow_up:
pkg/storage/s3/s3.go	79.79% <100.00%> (-0.17%)	:arrow_down:
... and 2 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[rchincha]

This PR is awaiting benchmark results. Both latency and throughput.

[andaaron]

This PR is awaiting benchmark results. Both latency and throughput.

This PR includes just adding an interface and refactoring boltdb. It doesn't include dynamodb related changes.

[chofnar]

Can you explain why we need this registering mechanism? Other drivers would be zot extensions with different build tags?

This is how distribution does it. I reckon it's the easiest way to register the available factories, by calling the Register function from the Driver's init function, since the Create func uses the same map to return the Drivers where they're needed

[rchincha]

There is a "pkg/api/*db" file in the PR - do you intend to merge this file? or was this for a test? if so, create/generate it from the Makefile or some setup routine.

[rchincha]

The code refactor is not very clear. I think we want to say:

1. zot has two major "states" - datapath blobs and metadata (which we call cache) for dedupe, gc, user-prefs etc
2. zot creates a cache interface like so: Create(name, driverName) -> cache
3. All "cache" instances implement the same api except over different backends.

For 2), put the interface under pkg/storage/cache.go, and move implementations out into their own files. I think it is scattered currently around cachedb.go etc.

For reference, look at pkg/storage/storage.go

[chofnar]

Commit ID with the caching on subpaths functionality: 49e4464a8461b6ed64a001910bdadcde2970297f

[andaaron]

There are some changes to be made which didn't get into the PR yet (move config validation in pkg/cli/root.go, make sure each image store has it's own blob cache to avoid hardlink attempts between blobs in different substores)