rusqlite
Add `wasm32-unknown-unknown` support
All credit to @passchaos in #935 for the insight that SQLite only uses a very small subset of libc, that can easily be copied in-tree.
Significant changes from that PR:
- Using pristine libc
.cfiles from OpenBSD - Minimal libc headers
-
libsqlite3-sys/sqlite3/sqlite3.ckept pristine - Removed custom
memvfs, replaced by a thin VFS that liberally returnsSQLITE_IOERR. This target is intended to be used only withConnection::open_in_memory()or equivalent. - Builds and runs a basic integration test in CI, via
wasm-pack.
~~The allocator's free() method is still a little weird, but it is documented that SQLite's default allocator stores the allocation length in the first 8 bytes: https://sqlite.org/malloc.html#the_default_memory_allocator . If we want to wrap it again, I'm happy to do that, just let me know.~~
To do:
- [x] Audit SQLite API for structures passed by value
- [x] Improve time-related and randomness implementations
Closes #488, closes #827, closes #935
Codecov Report
Attention: 89 lines in your changes are missing coverage. Please review.
Comparison is base (
a96f61b) 81.31% compared to head (415cafe) 80.62%.
| Files | Patch % | Lines |
|---|---|---|
| tests/wasm_pack.rs | 0.00% | 45 Missing :warning: |
| libsqlite3-sys/build.rs | 6.38% | 44 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #1010 +/- ##
==========================================
- Coverage 81.31% 80.62% -0.69%
==========================================
Files 51 52 +1
Lines 10402 10493 +91
==========================================
+ Hits 8458 8460 +2
- Misses 1944 2033 +89
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
I reviewed https://sqlite.org/capi3ref.html, and did not see any instances of structs being passed or returned by value.
@thomcc Does this PR look roughly acceptable, if I sort out the time and randomness-related bits?
Give me some time to review it (probably sometime over the weekend), but thanks for the work.
@wngr Thanks! It does seem plausible to make a VFS that uses IndexedDB as a backend; that would be very cool. I don't expect to personally have time to work on it in the near future, but would love to see it happen.
A concern from the previous PR:
A lot of the declarations in the header don't have definitions. What's the deal there?
I believe what's happening (in the few cases of this that still exist) is that the amalgamation is relying on certain declarations existing, but never actually uses them in the ultimately-generated code. If they were used, the WASM file would fail to load with its cryptic version of an "undefined symbol" error, see for example https://github.com/rustwasm/wasm-bindgen/issues/2570
This is essentially how I decided which libc files to include; provide the minimal declarations that allowed compilation, and then track down the symbols that needed to be added that would allow the WASM to load.
Edit: Also, it seems clang provides builtins for memcpy & friends.
@thomcc Ready for review.
- Reimplemented the allocator functions to explicitly store allocation size in a header.
- Implemented required time & randomness functions.
@thomcc Any chance you could review this PR? wasm32-unknown-unknown support would be awesome :smile:
I tried using this and ran into the following (click to expand)
error: failed to run custom build command for `libsqlite3-sys v0.23.2 (https://github.com/trevyn/rusqlite.git?branch=wasm32-unknown-unknown#004cf263)`
Caused by:
process didn't exit successfully: `/Users/py/projects/codebase-labs/ic-sqlite/main/target/debug/build/libsqlite3-sys-5deb3527a4482ea4/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-changed=sqlite3/sqlite3.c
cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
TARGET = Some("wasm32-unknown-unknown")
OPT_LEVEL = Some("0")
HOST = Some("aarch64-apple-darwin")
CC_wasm32-unknown-unknown = None
CC_wasm32_unknown_unknown = None
TARGET_CC = Some("cc")
CFLAGS_wasm32-unknown-unknown = None
CFLAGS_wasm32_unknown_unknown = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("llvm14-builtins-abi")
running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/Users/py/projects/codebase-labs/ic-sqlite/main/target/wasm32-unknown-unknown/debug/build/libsqlite3-sys-f5f455669bc46cf2/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c"
cargo:warning=In file included from sqlite3/sqlite3.c:24572:
cargo:warning=In file included from /nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:83:
cargo:warning=In file included from /nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/ucred.h:76:
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/bsm/audit.h:203:2: error: unknown type name 'dev_t'
cargo:warning= dev_t port;
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/bsm/audit.h:209:2: error: unknown type name 'dev_t'
cargo:warning= dev_t at_port;
cargo:warning= ^
cargo:warning=In file included from sqlite3/sqlite3.c:24572:
cargo:warning=In file included from /nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:83:
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/ucred.h:101:2: error: unknown type name 'u_int'
cargo:warning= u_int cr_version; /* structure layout version */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/ucred.h:104:2: error: unknown type name 'gid_t'
cargo:warning= gid_t cr_groups[NGROUPS]; /* advisory group list */
cargo:warning= ^
cargo:warning=In file included from sqlite3/sqlite3.c:24572:
cargo:warning=In file included from /nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:84:
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:109:2: error: unknown type name 'caddr_t'
cargo:warning= caddr_t user_stack; /* where user stack was allocated */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:114:2: error: unknown type name 'u_int'
cargo:warning= u_int p_estcpu; /* Time averaged value of p_cpticks. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:116:2: error: unknown type name 'fixpt_t'
cargo:warning= fixpt_t p_pctcpu; /* %cpu for this process during p_swtime */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:119:2: error: unknown type name 'u_int'
cargo:warning= u_int p_swtime; /* Time swapped in or out. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:120:2: error: unknown type name 'u_int'
cargo:warning= u_int p_slptime; /* Time since last blocked. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:123:2: error: unknown type name 'u_quad_t'
cargo:warning= u_quad_t p_uticks; /* Statclock hits in user mode. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:124:2: error: unknown type name 'u_quad_t'
cargo:warning= u_quad_t p_sticks; /* Statclock hits in system mode. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:125:2: error: unknown type name 'u_quad_t'
cargo:warning= u_quad_t p_iticks; /* Statclock hits processing intr. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:134:2: error: unknown type name 'u_char'; did you mean 'char'?
cargo:warning= u_char p_priority; /* Process priority. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:135:2: error: unknown type name 'u_char'; did you mean 'char'?
cargo:warning= u_char p_usrpri; /* User-priority based on p_cpu and p_nice. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:140:2: error: unknown type name 'u_short'; did you mean 'short'?
cargo:warning= u_short p_xstat; /* Exit status for wait; also stop signal. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/proc.h:141:2: error: unknown type name 'u_short'; did you mean 'short'?
cargo:warning= u_short p_acflag; /* Accounting flags. */
cargo:warning= ^
cargo:warning=In file included from sqlite3/sqlite3.c:24572:
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:453:2: error: unknown type name 'gid_t'
cargo:warning= gid_t p_rgid; /* Real group id. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:454:2: error: unknown type name 'gid_t'
cargo:warning= gid_t p_svgid; /* Saved effective group id. */
cargo:warning= ^
cargo:warning=/nix/store/qzzicx3cjj00z0c4kx3v0919gqp40mnp-libSystem-11.0.0/include/sys/sysctl.h:462:2: error: unknown type name 'gid_t'
cargo:warning= gid_t cr_groups[NGROUPS]; /* groups */
cargo:warning= ^
cargo:warning=fatal error: too many errors emitted, stopping now [-ferror-limit=]
cargo:warning=20 errors generated.
exit status: 1
--- stderr
error occurred: Command "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/Users/py/projects/codebase-labs/ic-sqlite/main/target/wasm32-unknown-unknown/debug/build/libsqlite3-sys-f5f455669bc46cf2/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c" with args "cc" did not execute successfully (status code exit status: 1).
I was previously trying to use @rkusa's branch and had to set TARGET_CC for that to build, but that's the only thing I'm explicitly setting here.
For what it's worth, this is what happens if I don't set TARGET_CC:
error: failed to run custom build command for `libsqlite3-sys v0.23.2 (https://github.com/trevyn/rusqlite.git?branch=wasm32-unknown-unknown#004cf263)`
Caused by:
process didn't exit successfully: `/Users/py/projects/codebase-labs/ic-sqlite/main/target/debug/build/libsqlite3-sys-5deb3527a4482ea4/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-changed=sqlite3/sqlite3.c
cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
TARGET = Some("wasm32-unknown-unknown")
OPT_LEVEL = Some("0")
HOST = Some("aarch64-apple-darwin")
CC_wasm32-unknown-unknown = None
CC_wasm32_unknown_unknown = None
TARGET_CC = None
CC = Some("clang")
CFLAGS_wasm32-unknown-unknown = None
CFLAGS_wasm32_unknown_unknown = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/Users/py/projects/codebase-labs/ic-sqlite/main/target/wasm32-unknown-unknown/debug/build/libsqlite3-sys-f5f455669bc46cf2/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c"
cargo:warning=clang-11: warning: argument unused during compilation: '-mmacos-version-min=11.0' [-Wunused-command-line-argument]
cargo:warning=clang-11: warning: argument unused during compilation: '-arch arm64' [-Wunused-command-line-argument]
cargo:warning=error: unknown target CPU 'apple-a13'
cargo:warning=note: valid target CPU values are: mvp, bleeding-edge, generic
exit status: 1
--- stderr
error occurred: Command "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/Users/py/projects/codebase-labs/ic-sqlite/main/target/wasm32-unknown-unknown/debug/build/libsqlite3-sys-f5f455669bc46cf2/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c" with args "clang" did not execute successfully (status code exit status: 1).
@paulyoung For an aarch64-apple-darwin host, this relies on a Homebrew clang install, it looks like yours might be out of date? brew install llvm might help.
Anyway, this still works for me with:
cargo install cargo-generate
cargo generate --git https://github.com/rustwasm/wasm-pack-template
rusqlite = {git = "https://github.com/trevyn/rusqlite", branch = "wasm32-unknown-unknown", features = ["bundled"]}
wasm-pack build
Does this work for you? Are you able to link to the project that you're actually building?
@trevyn I manage my projects and their dependencies using nix (not homebrew) so it's probably a matter of figuring out what versions are needed and making sure they're being used.
I could create a minimal example project that exhibits this issue if it would help.
I created a standalone project here:
https://github.com/paulyoung/rusqlite-wasm32-unknown-unknown-nix
and asked about it on the Nix forums here:
https://discourse.nixos.org/t/help-building-rusqlite-for-wasm32-unknown-unknown-with-crane/21724
Fancy! In that case, libsqlite3-sys/build.rs is looking for the CC and AR env variables to be set to a clang and llvm-ar that support wasm32-unknown-unknown. I know Homebrew's [email protected] works, not sure what the minimum requirements are otherwise.
https://github.com/trevyn/rusqlite/blob/004cf263143e038f01e6022d8d5b81b9c621f89f/libsqlite3-sys/build.rs#L253-L261
This built fine on my m1 mac, then it failed on docker: rust:1.69.0-bookworm until i installed clang after which it built fine.
Would love to see this merged into master 🎉
I think that this issue its related https://github.com/rustwasm/wasm-bindgen/issues/3454
I just wrote a test to init a DB connection and INSERT then SELECT using this branch (with latest master pulled into it).
It runs fine on non-wasm but when I ran it on wasm it throws: RuntimeError: indirect call to null. I could repro this on firefox and safari. The full stacktrace is here:
JS exception that was thrown:
RuntimeError: indirect call to null
sqlite3OsFileControl@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3403]:0x3340f3
sqlite3_file_control@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[761]:0x20d9e1
sqlite3Pragma@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[61]:0x26e6d
yy_reduce@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[60]:0x2454c
sqlite3Parser@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[971]:0x23e22e
sqlite3RunParser@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[301]:0x15b901
sqlite3Prepare@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[277]:0x14cafc
sqlite3LockAndPrepare@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[1029]:0x249b15
sqlite3_prepare_v3@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3523]:0x339c56
rusqlite::inner_connection::InnerConnection::prepare_::hc2f0a88ee06fb549@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[4084]:0x34f87a
rusqlite::inner_connection::InnerConnection::prepare::h94bf9f3edeb66287@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[427]:0x19d8a3
rusqlite::Connection::prepare_with_flags::h09f3557df09b0fc9@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3308]:0x32f292
rusqlite::Connection::prepare::h310e550dfb17c830@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[4451]:0x35a312
rusqlite::Connection::execute_batch::hea1042c16c0cd981@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[294]:0x1570c0
rusqlite::pragma::<impl rusqlite::Connection>::pragma_update::hdec1666d01ace9c7@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[621]:0x1e5e75
rusqlite_migration::set_user_version::h982001b4ab2782a6@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[1417]:0x28b38a
rusqlite_migration::Migrations::goto_up::h727ad997790a6cb9@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[154]:0xe7555
rusqlite_migration::Migrations::goto::h1a94cc5b0cbd3f29@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[207]:0x119bcf
rusqlite_migration::Migrations::to_latest::hcecaa0df0427a54b@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[705]:0x1febfe
db::lib_test::test::hc7c7c3310ec12cbf@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[628]:0x1e7f23
core::ops::function::FnOnce::call_once::h8dd7fba8693501e0@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[5668]:0x370078
wasm_bindgen_test::__rt::Context::execute_sync::{{closure}}::he71cc68df0d3c2a7@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3323]:0x32ff13
<wasm_bindgen_test::__rt::TestFuture<F> as core::future::future::Future>::poll::{{closure}}::{{closure}}::hc25b673a04f73bb8@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3518]:0x33987e
wasm_bindgen::convert::closures::invoke0_mut::h6e1e9c6b868d2ee7@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[4063]:0x34ed5a
__wbg_adapter_41@http://127.0.0.1:38265/wasm-bindgen-test:337:10
cb0@http://127.0.0.1:38265/wasm-bindgen-test:487:28
window.__wbg_test_invoke@http://127.0.0.1:38265/:38:38
__wbg_get_imports/imports.wbg.__wbg_wbgtestinvoke_7de5d5374a833823/<@http://127.0.0.1:38265/wasm-bindgen-test:492:30
handleError@http://127.0.0.1:38265/wasm-bindgen-test:342:18
__wbg_get_imports/imports.wbg.__wbg_wbgtestinvoke_7de5d5374a833823@http://127.0.0.1:38265/wasm-bindgen-test:480:76
wasm_bindgen_test::__rt::__wbg_test_invoke::hb58221eb1bd2a29f@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[2517]:0x2fdf69
<wasm_bindgen_test::__rt::TestFuture<F> as core::future::future::Future>::poll::{{closure}}::hf3411908db1aeefa@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3839]:0x346df8
scoped_tls::ScopedKey<T>::set::h73e8dcd9e422f126@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3629]:0x33e690
<wasm_bindgen_test::__rt::TestFuture<F> as core::future::future::Future>::poll::h4925da01429b92e6@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[653]:0x1ef992
<wasm_bindgen_test::__rt::ExecuteTests as core::future::future::Future>::poll::h086a35e5c805ca10@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[184]:0x104fcd
wasm_bindgen_test::__rt::Context::run::{{closure}}::hc9ef03a3fb10a54c@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[1576]:0x2a0982
wasm_bindgen_futures::future_to_promise::{{closure}}::{{closure}}::h32d6024e9975e335@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[714]:0x20109e
wasm_bindgen_futures::task::singlethread::Task::run::h184e97cc0ef1e77e@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[1510]:0x297fe8
wasm_bindgen_futures::queue::QueueState::run_all::h4285a3205cd25e58@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[1151]:0x260bb3
wasm_bindgen_futures::queue::Queue::new::{{closure}}::hc990aef658832399@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[4229]:0x35407f
<dyn core::ops::function::FnMut<(A,)>+Output = R as wasm_bindgen::closure::WasmClosure>::describe::invoke::h17f3b0b306d2221f@http://127.0.0.1:38265/wasm-bindgen-test_bg.wasm:wasm-function[3349]:0x33156b
__wbg_adapter_28@http://127.0.0.1:38265/wasm-bindgen-test:240:10
real@http://127.0.0.1:38265/wasm-bindgen-test:203:20
where the test is:
use wasm_bindgen_test::{wasm_bindgen_test, wasm_bindgen_test_configure};
wasm_bindgen_test_configure!(run_in_browser);
#[derive(Debug, Eq, PartialEq, Clone)]
struct Example {
pub id: i64,
pub name: String,
}
#[wasm_bindgen_test]
#[test]
fn test() {
let migrations = rusqlite_migration::Migrations::new(vec![rusqlite_migration::M::up(
"CREATE TABLE example (id INT, name TEXT);",
)]);
let mut conn = rusqlite::Connection::open_in_memory().unwrap();
migrations.to_latest(&mut conn).unwrap();
let val = Example {
id: 1,
name: "test".into(),
};
conn.execute(
"INSERT INTO example (id, name) VALUES (?, ?)",
(val.id, val.name.clone()),
)
.expect("Failed to set");
let res = conn
.query_row(
"SELECT id, name FROM example WHERE (id=?)",
[val.id],
|row| {
Ok(Example {
id: row.get(0).unwrap(),
name: row.get(1).unwrap(),
})
},
)
.unwrap();
assert_eq!(res, val);
}
Ok, it looks like this is totally broken with the latest SQLite; they've done something with long double constants that make clang bring in the f128 builtins even with -DLONGDOUBLE_TYPE=float. The builtins need the clang compiler-rt runtime linked in, and this is a bit of a nightmare with Rust and wasm.
It also looks like sqlite added a dep on strspn (https://github.com/sqlite/sqlite/commit/028acd974f3c03c489dfaa02da618af99613cde7#diff-40aa21688f1ac58ddf8ef5f0f4e472e6075411f5c4f75b9c1d512b7e03a2cbabR849). I assume this needs to be added to the libc shim in https://github.com/trevyn/rusqlite/tree/wasm32-unknown-unknown/libsqlite3-sys/sqlite3/wasm32-unknown-unknown/libc?
As for the new deps on various long double routines, the only ones that are currently pulled in are:
(import "env" "__extenddftf2" (func $__extenddftf2 (type 99)))
(import "env" "__getf2" (func $__getf2 (type 123)))
(import "env" "__multf3" (func $__multf3 (type 94)))
(import "env" "__trunctfdf2" (func $__trunctfdf2 (type 118)))
(import "env" "__lttf2" (func $__lttf2 (type 123)))
(import "env" "__gttf2" (func $__gttf2 (type 123)))
I guess the official sqlite3 source hasn't needed to worry about this since AFAIK they target emscripten and it apparently includes all of compiler-rt (https://github.com/emscripten-core/emscripten/tree/4b33521f103c2fb4fb218b1e1c8637a4a96b8e74/system/lib/compiler-rt)???
I'll ask around and see if anyone has figured this out yet.
I would love to see this! Can I somehow help?
So the current status is that we need to resolve the implementations of the 6 builtins carlsverre mentioned.
I can think of a few ways, in ascending order of suitability (from my perspective):
-
Bring in a compiled library for the builtins e.g. https://github.com/jedisct1/libclang_rt.builtins-wasm32.a. This appears to be non-trivial to build from scratch, and I don't think adding a binary blob to the rusqlite repo is a great idea.
-
Report the issue to the SQLite team; the change that makes clang bring in long double routines was recent, and it seems they were willing to make a fix for a somewhat related issue: https://sqlite.org/src/info/legacy-msvc-workaround. I don't have a pointer to the exact change that caused the issue, or I would have made a casual post myself.
-
Bring in C implementations from https://github.com/llvm/llvm-project/tree/main/compiler-rt/lib/builtins, and copy them in-tree as I did for the necessary libc routines.
-
Interestingly, Rust just merged the RFC for
f128support, and an implementation of that would likely include implementations of the builtins in question into https://github.com/rust-lang/compiler-builtins. (Which I believe would requiref128support in the compiler first.) See https://github.com/rust-lang/rust/issues/116909
I think moving any of these forward would be helpful.
@trevyn I was snooping around the sqlite changelog and found this gem in the 3.44.0 release:
13. SQLite now performs run-time detection of whether or not the underlying hardware supports "long double" with precision greater than "double" and uses appropriate floating-point routines depending on what it discovered.
It appears that your current rusqlite branch with wasm support is using sqlite 3.43.2. Is it easy to update that and see if 3.44.0 fixes the problem?
Thanks for testing it. Didn't realize they are just performing run-time checks.
I did some research and have discovered that the wasi-sdk project releases libclang_rt as a standalone download here: https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-21
Upon inspecting that file, it's an ar archive containing a bunch of wasm objects that appear mostly ready to link. Out of the f128 functions I identified above as missing, this archive contains implementations of every function apart from __lttf2 and __gttf2. It seems that clang uses __letf2 and __getf2 instead.
It doesn't seem trivial to link that archive in during the build though in a way that correctly links all of those functions. But it's perhaps one possible way to move forward.
Oh, just noticed that is the same file as https://github.com/jedisct1/libclang_rt.builtins-wasm32.a which you mentioned in https://github.com/rusqlite/rusqlite/pull/1010#issuecomment-1774097822... Apologies.
Ok after more testing, I can confirm that the following diff resolves all of the missing imports:
diff --git a/libsqlite3-sys/build.rs b/libsqlite3-sys/build.rs
index a6b99c3..55a7488 100644
--- a/libsqlite3-sys/build.rs
+++ b/libsqlite3-sys/build.rs
@@ -291,6 +291,12 @@ mod build_bundled {
cfg.file("sqlite3/wasm32-unknown-unknown/libc/string/strncmp.c");
cfg.file("sqlite3/wasm32-unknown-unknown/libc/string/strrchr.c");
cfg.file("sqlite3/wasm32-unknown-unknown/libc/string/strspn.c");
+
+ cfg.file("/home/carl/dev/llvm-project/compiler-rt/lib/builtins/extenddftf2.c");
+ cfg.file("/home/carl/dev/llvm-project/compiler-rt/lib/builtins/comparetf2.c");
+ cfg.file("/home/carl/dev/llvm-project/compiler-rt/lib/builtins/multf3.c");
+ cfg.file("/home/carl/dev/llvm-project/compiler-rt/lib/builtins/trunctfdf2.c");
}
if cfg!(feature = "unlock_notify") {
cfg.flag("-DSQLITE_ENABLE_UNLOCK_NOTIFY");
@trevyn I got something working here: https://github.com/trevyn/rusqlite/pull/1