WiseLibs
Issues distributing a node CLI that uses better-sqlite3 as a dependency
Claude Code currently packages better-sqlite3 as an external dependency. Some users are reporting issues here: https://github.com/anthropics/claude-code/issues/908
It seems like many users are able to run without issue, but some are stuck.
It's possible that we are packaging this incorrectly.
I'm able to break things by going to node 18.3.0 from a later version
When the dependency is broken, npm rebuild better-sqlite3 doesn't work if I have a python venv active
It works if it is not active.
Because we are a dev tool and are very attached to auto-update, telling folks to deactivate their venv is not practical for us.
Hope we can get some advice or help!
Well, better-sqlite3 is not actually a part of the package but only referenced as a required dependency. This will cause the automatics of prebuild-install to take over and try to find a suitable binary (from the available releases) for the used runtime, when installing your module.
When it comes down to NodeJS, there are prebuilt binaries for NodeJS 18, 20, 22 and 23 (please note, NodeJS 18 reached it EoL on 30th of April). As long as one of these versions and npm is used, installing better-sqlite3 as a required dependency should work.
If a prebuilt binary could not be resolved, a recompilation is attempted / required. This might be the case when using Bun instead of NodeJS. This step relies on having a proper setup of Python + compiler to run.
If pnpm is used instead of npm for installing, this adds additional challenges due to it's internal treatment of dependencies and caching which may require manual actions, as your users already found out.
I fear there is no easy way to address all those issues and scenarios at once. If you really wanted to be sure, that at least a binary of better-sqlite3 is contained within the package itself, you would have to include all the different prebuild binaries for the desired versions of NodeJS and architectures. In addition, you would have to setup some custom resolving and loading of the native module, but as you also have drizzle in the setup, this might not be that trivial.
In the meantime, what can end users actually do to fix this?
The errors seems to have been present in this project for a number of years, so an entry in the README with some suggestions would probably be appreciated by a variety of users.
Cheers - V
These are not actual errors, but side effects based on user setups, which are not covered by the setup toolchain for prebuilt binaries.
As better-sqlite3 is a native module, it requires to be compatible with the used runtime and OS to be loaded. This requires compilation fo the C code of SQLite. To skip the compilation for "well-known" setups, prebuild-install and prebuild are used to cover builds for NodeJS (18, 20, 22, 23) and Electron (26 and up) for the most common OS / architectures.
In order to "fix the errors", users should stick to a runtime and OS covered by the prebuilt binaries. Also, from my personal experience, npm or yarn should be used instead of pnpm, if one does not want to adjust pnpm wo work correctly with native modules.
And yes, I too think the README and troubleshooting docs could be improved regarding these install issues, when not using a well-known setup.
This is such a lazy answer. The real issue is that the entire process of versioning and how binaries are distributed are simply ridiculous and embarassing.
@wolffiex the only solution I'm aware of (for literally any software) is to tell your users what specific versions of official node runtimes, OS platforms, architectures, and only npm are supported setups.
There are at least 3 node.js-like runtimes, and at least 4 npm alternatives--all of which behave substantively differently than vanilla node.js and npm.
"But it doesn't work with bun/deno/pnpm/bsd!"
Correct! Pull requests are welcome.
The issue referenced in the title of this issue was fixed a long time ago. We are in the process of switching to a single binary executable, which is the solution to these problems. If you want to try it, we'd love your feedback. See here: https://docs.anthropic.com/en/docs/claude-code/setup#native-binary-installation-alpha