Package duckDB extension alongwith rill binary

[nishantmonu51]

At present we download the binaries when starting rill, which can fail if duckDB CDN is having issues.

HTTP Error: Failed to download extension "json" at URL "http://nightly-extensions.duckdb.org/v0.9.2/osx_arm64/json.duckdb_extension.gz" Extension "json" is an existing extension.

We would like to avoid downloading the extensions at runtime.

[nishantmonu51]

@himadrisingh : Can you get this assigned on Infra side and prioritize it ?

[himadrisingh]

This will increase the size of the cli though?

[begelundmuller]

Implementation proposal:

• To maintain a single binary, we need to use Go's embed package to embed the DuckDB extensions we use in the binary. See runtime/pkg/examples for an example of how we use it to embed example projects.
• We should implement the embedding in our DuckDB driver (runtime/drivers/duckdb) and unpack the embedded extensions before first opening a DuckDB handle.
  • We should probably unpack to ~/.duckdb/extensions, which is DuckDB's default location for extensions (notice a specific nested folder structure is required).
  • We should avoid unpacking an extension if it already exists (for faster startup after the first run).
• We also need a script that downloads the DuckDB extensions we rely on and places them in the embed directory (probably runtime/drivers/duckdb/embed). The script could be placed in the scripts directory and invoked when make cli is executed (in the cli.prepare Make target)
• Lastly, we need to embed different DuckDB extensions for different architectures (they are architecture specific). Since we build for multiple architectures using a cross-compiler, we can't just download extensions for the current system architecture. There's a few ways to solve this issue – here's one proposal:
  • Have the download script download extensions for all supported architectures and place them in runtime/drivers/duckdb/embed/<arch>/<duckdb version>/
  • Use Go build tags to only embed extensions for the current architecture. For example, runtime/drivers/duckdb/embed_darwin_arm64.go would have the build tag //go:build darwin && arm64 and embed ./embed/darwin_arm64/**