opteryx
opteryx copied to clipboard
mabel-dev
🦖 A SQL-on-everything Query Engine you can execute over multiple databases and file formats. Query your data, where it lives.
### Thank you for taking the time to report a problem with Opteryx. _To help us to respond to your request we ask that you try to provide the below...
Support non-equi joins. This is likely to have one of two implementations (or both and let the optimizer work out for each instance): - rewrite as CROSS JOIN and filter...
Rules - parquet files, row width 2048 bytes, filter on a 16 character string attribute in jsonb field - parquet files with 50k rows - equals filter, matching about 10...
### Thank you for taking the time to report a problem with Opteryx. _To help us to respond to your request we ask that you try to provide the below...
There's a Row loader which just takes a raw Tuple and doesn't need any further processing
Perform some parts of the Row load and read using Cython to improve performance. ~~~python from cpython.tuple cimport PyTupleObject from libc.string cimport strcmp cdef class ExtendedTuple: cdef PyTupleObject* values cdef...
Initially, we're mainly testing performance compared to Parquet - the goal is to be at least 25% faster than zstd parquet files (the current fastest format) if a completely contrived...
~~~python import pyarrow as pa def write_arrow_to_bytes(data: pa.Table) -> bytes: """ Serialize a PyArrow Table to a bytes array. Parameters: data: pa.Table The PyArrow Table to serialize. Returns: bytes The...
The binder shouldn't be concerned with the caching functionality, the readers should attach the caching (the decorator) and the decorator should determine if it should do the work. This will...
- [ ] Daily batch update of NVD data - [ ] High frequency batch - https://tfl.gov.uk/info-for/open-data-users/open-data-policy - [ ] streaming - sensor data