sled
sled copied to clipboard
spacejam
Provide CFFI/PyO3 bindings for Python
The current Python bindings use ctypes which are not necessary on CPython and are slow on PyPy. In addition, the package in it's current form cannot be released to PyPi which is a shame. PyO3 provides a way to write CPython extensions. These extensions can be released as wheels which everyone can use. On PyPy CFFI is more efficient so we can provide CFFI bindings as well although PyO3 supports PyPy through PyPy's slower cpyext compatibility layer. Since the API surface of sled is small, a CFFI extension could be enough for both PyPy and CPython.
I'm willing to help in this regard if there's interest in it.
Hey @thedrow thanks for the info! Python was my first "professional" language that I used at work almost 10 years ago but I never got into the FFI side of things, so I really appreciate the perspective on this.
How big is the performance impact of using PyO3 from pypy? If you can do 1 million function calls through it per second on one thread then that's more than enough for the current state of sled, and I'd prefer to minimize extra code unless it can significantly improve performance on a real use case.
Well the answer to that question is tricky. We'll need to write that extension and measure. However, CFFI should be enough to implement the wrapper you wrote using ctypes. Is there something missing from the bindings currently?
I think it's probably best to start with the option that results in the least code that needs to be maintained, and from there measure how much of the overall cpu time is spent handling ffi to measure if it would be worth the time to implement the second option as well.
The FFI code has not been touched in over a year, and both sled-native and the ctypes implementation are quite out of date. The first thing to do is update sled-native to use the latest sled master branch. I think as long as it can run the basic insert and remove commands then that's a good initial step. I might have time to update sled-native at some point in the next 2 weeks, but if you'd like to tackle it I can help answer questions, ideally on the discord chat to minimize latency but in this issue is fine too if you would prefer.
Hi, I'd like to try Sled on a python project for parallel writes from multiple processes (very tricky with RocksDB). Is there an update on this issue?
