buildkit
buildkit copied to clipboard
solver: add sqlcachestorage as an alternative to the bolt db
This creates an sql cache storage that stores the cache key index in an sqlite3 database instead of in bolt. At the moment, this is just functional in the bare minimum as a way to test the feasibility of this method. There are ways to configure the sqlite database for more efficient transactions.
This can be used as an alternative storage for bolt db and is potentially usable as a distributed cache key storage for another implementation in the future.
I thought I'd experiment with this idea because I had noticed that the cachedb had a relational pattern. I haven't done any profiling with this, but I did note some things while working on this PR.
- sqlite specifically has a WAL journaling mode which could be a performance booster. We could test the feasibility of the WAL being in a temporary storage or native disk.
- With SQL, it's also possible to move the cache index off a local disk to begin with. We might want to batch transactions some amount though.
- The
_backlinksand_byresultbuckets from bolt weren't needed since they were just custom index implementations. This implementation just creates an index and lets SQL handle that. - The logic for accessing the database is, in general, more simple in my own opinion because the SQL statements that query and modify it are mostly self-explanatory.
- I think that there are some additional changes that could be made to
solver/cachemanager.goto do some of the logic directly in the database rather than throughWalkstatements.Walkstatements aren't the biggest deal when the cachedb is local, but incurring a network round trip can be more difficult if we want to make it easy to utilize client/server models for the database. In particular,getIDFromDepsis something I believe that can be rewritten with an SQL query.