TileDB-Py
TileDB-Py copied to clipboard
TileDB-Inc
Asynchronous Writes/Reads
This feature is currently only available in c++. Any chance somebody could write the python wrapper for this in the near future ?
If not, dispatching the write operation in a new thread should work fine, shoudn't it ?
All the IO / cpu intensive operations do give up the GIL, so usually the best option is to dispatch the write on another thread and then get a future to that (blocking) operation.
Full async support (integration with the event loop), probably should follow after making the above operation (which you can do now) a bit more user friendly and documented.
Full async support (integration with the event loop), probably should follow after making the above operation (which you can do now) a bit more user friendly and documented.
Hi @jakebolewski - is this too far away still? I'm happy to code some manual stuff but some documentation with any idiosyncrasies around the c++/python api would help.
PS: I'm available to help since I got an ongoing related project that would benefit from this.
Hi @ocehugo, are you looking specifically for integration with Python 3's async/await? At some point I looked in to this and saw that Cython doesn't support the async annotation on cdef functions, but I'll take a look again soon to try to work around that (it's not clear to me how strict the interpreter is about functions defined in C -- at worst we can probably hack in the attribute it wants).
Pull-requests very welcome of course! 😄
@ihnorton yeap - python 3.6+ probably, since some breaking API changes happened before that.
I'm happy to push something but I'm not familiar with the API internals and the C++ code. I'm just a very ordinary user ATM. Im a bit rust in cython but played with it before. If you could get me some direction or where to look for the core things It can help a lot.
FYI: My main objective is to do Async fetching on object storage, mostly dense gridded stuff and associated metadata with time-x-y ranges.