SQLite alternatives

[guyfedwards]

Carrying on from discussion on https://github.com/guyfedwards/nom/issues/155.

While I understand why using a sqlite in-memory database is an attractive solution, I would like to suggest instead writing a simple, pure-go in-memory store. Why? Because I am selfish and I am trying to make my life easier 😄.

I have a fork that replaces sqlite with badger, primarily so that we can avoid the CGO dependency when building container images. With a pure go solution, the image doesn't need to include anything other than the nom binary; it doesn't require a full set of shared libraries, etc.

It's not ready for prime time (in particular, it doesn't include any facility for migrating from a sqlite store, which is why I haven't made a pr yet), but by keeping our sqlite dependency isolated in sqlitestore, we make this sort of thing easier to implement.

I have a alternative in-memory store implementation built on top of basic go data structures in https://github.com/larsks/nom/compare/feature/memory-store; it makes the follow changes:

Move the sqlite based store to internal/store/sqlitestore.
Add a new storage driver, internal/store/memorystore.

Note that this is built on top of some changes related to https://github.com/guyfedwards/nom/issues/167 (it doesn't depend on these changes, and I'd be happy to rebase things if folks are interested in this solution and want to avoid the GUID-based changes for now).

[guyfedwards]

Copying my comment here for continuity:

@larsks I'm not sure I see the benefit of moving to badger/another key/value store really. There are downsides of using CGO but I haven't seen them produce actual issues for users. If there was then I think looking at something like cznic/sqlite or ncruces/go-sqlite3 would be more likely, to maintain compatibility between versions.

[larsks]

There are downsides of using CGO but I haven't seen them produce actual issues for users

Again, my interest was primarily in removing dependencies from the container images. With a pure-Go build, our container image only needs to contain a single file. Since the actual backend is hidden behind the Store interface, replacing it is relatively easy.

I'm happy to look at the Go-based sqlite analogues.

[GV14982]

IIRC the go based SQLite implementations are generally compatible, but often times are slower.

[guyfedwards]

I'm not particularly tied to sqlite, and the CGO stuff was a PITA to setup but since it's working I don't know if the juice of less dependencies/easier build, is worth the squeeze of having to migrate users nom store to a new format.