Add the option to statically link extensions and disable further extension installation

[breckcs]

By default, DuckDB will install and load extensions at run-time. In environments that do not have access to the Internet, or in environments where extension loading must be strictly controlled for cybersecurity or compliance, DuckDB can 1) use a local extension repository, or 2) it can be compiled with the extensions statically linked.

1. The local extension repository can be set using the following DuckDB statement:

   set custom_extension_repository = 'my/path';
   

   The easiest way to seed this is to run DuckDB, install the extensions you want, then set the DuckDB extensions cache (e.g., ~/duckdb/extensions) to the extension repository path.

2. The other option is to compile DuckDB from source and statically link the extensions. From the DuckDB documentation on building extensions:

   git clone https://github.com/duckdb/duckdb.git
   cd duckdb
   git checkout v1.2.1
   CORE_EXTENSIONS='autocomplete;httpfs;icu;json;tpch' DISABLE_EXTENSION_LOAD=1 GEN=ninja make
   

   Furthermore, the DISABLE_EXTENSION_LOAD=1 will prevent the run-time installation of any additional extensions.

   Running DuckDB and reporting the extensions shows they are statically linked, and additional extensions cannot be loaded:

   % ./build/release/duckdb
   v1.2.1 8e52ec4395
   Enter ".help" for usage hints.
   Connected to a transient in-memory database.
   Use ".open FILENAME" to reopen on a persistent database.
   D SELECT * FROM duckdb_extensions();
   ┌──────────────────┬─────────┬───────────┬───┬───────────────────┬───────────────────┬────────────────┐
   │  extension_name  │ loaded  │ installed │ … │ extension_version │   install_mode    │ installed_from │
   │     varchar      │ boolean │  boolean  │   │      varchar      │      varchar      │    varchar     │
   ├──────────────────┼─────────┼───────────┼───┼───────────────────┼───────────────────┼────────────────┤
   │ arrow            │ false   │ false     │ … │                   │ NULL              │                │
   │ autocomplete     │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ aws              │ false   │ false     │ … │                   │ NULL              │                │
   │ azure            │ false   │ false     │ … │                   │ NULL              │                │
   │ core_functions   │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ delta            │ false   │ false     │ … │                   │ NULL              │                │
   │ excel            │ false   │ false     │ … │                   │ NULL              │                │
   │ fts              │ false   │ false     │ … │                   │ NULL              │                │
   │ httpfs           │ true    │ true      │ … │ 85ac466           │ STATICALLY_LINKED │                │
   │ iceberg          │ false   │ false     │ … │                   │ NULL              │                │
   │ icu              │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ inet             │ false   │ false     │ … │                   │ NULL              │                │
   │ jemalloc         │ false   │ false     │ … │                   │ NULL              │                │
   │ json             │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ motherduck       │ false   │ false     │ … │                   │ NULL              │                │
   │ mysql_scanner    │ false   │ false     │ … │                   │ NULL              │                │
   │ parquet          │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ postgres_scanner │ false   │ false     │ … │                   │ NULL              │                │
   │ shell            │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ spatial          │ false   │ false     │ … │                   │ NULL              │                │
   │ sqlite_scanner   │ false   │ false     │ … │                   │ NULL              │                │
   │ tpcds            │ false   │ false     │ … │                   │ NULL              │                │
   │ tpch             │ true    │ true      │ … │ v1.2.1            │ STATICALLY_LINKED │                │
   │ ui               │ false   │ false     │ … │                   │ NULL              │                │
   │ vss              │ false   │ false     │ … │                   │ NULL              │                │
   ├──────────────────┴─────────┴───────────┴───┴───────────────────┴───────────────────┴────────────────┤
   │ 25 rows                                                                         9 columns (6 shown) │
   └─────────────────────────────────────────────────────────────────────────────────────────────────────┘
   D INSTALL sqlite_scanner;
   Permission Error:
   Installing external extensions is disabled through a compile time flag
   D INSTALL sqlite_scanner;
   

I would like duckdb-rs to support this behaviour through Cargo.toml features.

It doesn't seem easy to do with the current approach of using the cc-rs crate in the crates/libduckdb-sys/build.rs file. It would likely be easier if the build.rs file used cmake or ninja-writer instead. An attempt has already been made by @njaard to change build.rs to use cmake, but it was never merged.

Is there any interest in supporting this feature?

[shuoli84]

Yea, this is really a pain when deploy duckdb in datacenter, the download phase caused first few queries way slow. Now I add a separate step to install extensions in Dockerfile. (I feel just thumbup emoji is not strong enough :))

[mlafeldt]

Hey @breckcs,

Thanks for the suggestion. I think using CMake directly would definitely be worth a shot to improve extension compatibility. Will take a closer look!

In the meantime, I recommend building libduckdb outside of duckdb-rs using DISABLE_EXTENSION_LOAD=1 and whatever else you need, and then link against the result without --features bundled.

[breckcs]

@mlafeldt, yes, that's what I'm doing.

I wrote about how I'm building libduckdb: Securing DuckDB, Improving Startup Time, and Working Offline.

Then I include duckdb-rs at the same version in Cargo.toml and omit the bundled feature.

This works, but it is tedious to do every release, and error prone to keep the versions aligned.

[lukesneeringer]

link against the result without --features bundled.

How does one do that?

[mlafeldt]

@lukesneeringer I added more detailed instructions for Linux and macOS in https://github.com/duckdb/duckdb-rs/pull/563