buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard

solver: add sqlcachestorage as an alternative to the bolt db

Open jsternberg opened this issue 1 year ago • 1 comments

This creates an sql cache storage that stores the cache key index in an sqlite3 database instead of in bolt. At the moment, this is just functional in the bare minimum as a way to test the feasibility of this method. There are ways to configure the sqlite database for more efficient transactions.

This can be used as an alternative storage for bolt db and is potentially usable as a distributed cache key storage for another implementation in the future.

jsternberg avatar Aug 12 '24 20:08 jsternberg

I thought I'd experiment with this idea because I had noticed that the cachedb had a relational pattern. I haven't done any profiling with this, but I did note some things while working on this PR.

  1. sqlite specifically has a WAL journaling mode which could be a performance booster. We could test the feasibility of the WAL being in a temporary storage or native disk.
  2. With SQL, it's also possible to move the cache index off a local disk to begin with. We might want to batch transactions some amount though.
  3. The _backlinks and _byresult buckets from bolt weren't needed since they were just custom index implementations. This implementation just creates an index and lets SQL handle that.
  4. The logic for accessing the database is, in general, more simple in my own opinion because the SQL statements that query and modify it are mostly self-explanatory.
  5. I think that there are some additional changes that could be made to solver/cachemanager.go to do some of the logic directly in the database rather than through Walk statements. Walk statements aren't the biggest deal when the cachedb is local, but incurring a network round trip can be more difficult if we want to make it easy to utilize client/server models for the database. In particular, getIDFromDeps is something I believe that can be rewritten with an SQL query.

jsternberg avatar Aug 12 '24 20:08 jsternberg