Reconsider using rust for extensions

[fire]

We intend to add Rust to implement new capabilities, but not exclusively so.

[glommer]

hello @fire ! I think this would be best stated as a Discussion (https://github.com/libsql/libsql/discussions)

But in the mean time, what would you suggest?

[fire]

I don't know, but it is a barrier to giving libsql the greatest reach. Also can be a discussion.

[penberg]

@fire Can you elaborate on what your concern is? I don't think Rust as a language is that controversial choice, even the Linux kernel has support for it now. We absolutely should not close the door for people writing extensions in C, C++, Zig, or other languages, but I don't necessarily see a problem with having Rust code in the tree?

[psarna]

The biggest issue for introducing Rust is that it threatens the "just a .c and .h file" integration model that sqlite has, and it's indeed very nice and simple. We'll need to be very careful when designing the first Rust addition, but I think that "just a .c, .rs and .h file" is not a huge stretch.

[Dessix]

That integration model may be outdated with modern package management; would better code organization be a consequence of Rust integration, and lead to a more maintainable project overall?

[llimllib]

One benefit of the C agglomeration file is that it makes makes compilation to wasm quite easy; here I show how to use it to compile an extension into the build.

I imagine that for apps which embed sqlite, the "one c and h file" is a great boon - perhaps libsql will not be intending to be easily embeddable, but it should be clear about it if so.

[glommer]

In my mind, keeping compatibility with the file format is 110% crucial. Keeping compatibility with the existing APIs is 100% crucial (but adding new APIs should be fine, doing good async on top of the existing APIs will be hard).

The rest is good to have, and we should keep if possible, but at some point not crucial.

Is the "one c file" help with embedding this in other applications that are not written in C?

[llimllib]

I think so, yes - C is kind of a lingua franca for embedding

[Dessix]

If fixed-file-set is critical to that degree, it's feasible to merge Rust directories into single files by translating files to mod scopes, as they aren't capable of cross-contamination of scope content. As for being a universal language, Rust is rapidly approaching that point, and will likely only improve with it's adoption by Linux.

If these are the priorities, I'd recommend using a C header to describe the Rust-extended contents alongside those exposed by the .c file, as suggested by @psarna above.

[glommer]

I am willing to be convinced, but for me, the interesting aspect of "embedded" is that you add attach to your application through an API, versus going through some remote call in a different process or network.

The one .c /.h file is one way to attach this to your application, but so is a .so library that can be statically or dynamically linked to your application.

I think it is premature to make the decision of which will be. There is an obvious benefit in keeping sqlite's way of embedding, and if and when it becomes a problem for integrating something else we can reevaluate.