use dynamic plugins instead of static, built-in modules

[naturallymitchell]

I've read some resources that say it can't be done, some that say rustc supports it but cargo doesn't, and others that say it's totally supported.

it seems available, and it would make tantivy development easier... (performance dropped 50% with it built-in, and another 50% with decent usage in lighttouch, so we did cfg feature)

I'm also worried about performance continuing to degrade, or suddenly degrading significantly, unexpectedly from anyone of these libraries due to some unexpected complexity, like with tantivy

this would also make development a lot faster with a package manager including plugins rather than depending on this project for everything

https://news.ycombinator.com/item?id=16736725 https://michael-f-bryan.github.io/rust-ffi-guide/dynamic_loading.html#lets-make-a-plugin rust-lang/cargo/issues/4538

[dariusc93]

It is possible to write plugins in rust using FFI which can be used for other languages (and technically can be use for rust when calling dlopen or using a crate that would use that or similar calls). The problem is that youre having to write things specifically for this to work correctly since rust ABI isnt stable to where one could write shared libraries that can be loaded dynamically in rust without needing to go this extra step. It should be noted that it could add more maintenance since one would likely be diving more into unsafe code which can introduce bugs, memory leaks, etc. Dynamic libraries dont always mean better performance (though it does have a lower overhead) since they are more complex than one would think, and dynamic libraries dont always introduce true portability since they would have to be compiled for each platform thus adding more maintenance when dealing with it between eg linux and windows. On top of that, on linux since we are using musl and assuming musl do support dyn libs, each library would likely have a large file size even in release due to it statically linking libraries and instead of them being linked into one, they would be linked to each library created thus increasing the size of the library and the likelihood of a large memory usage when loading the libraries.

[naturallymitchell]

https://nullderef.com/series/rust-plugins/

https://www.reddit.com/r/rust/comments/sq4r1i/plugins_in_rust_getting_our_hands_dirty/