spiraldb
Clean up alignment
We need to decide to what level of fidelity we want to enforce alignment. Currently, we enforce nothing but then occasionally run into interop issues with Arrow. To work around these, we ensure that Arrow does the allocation (e.g., on the VarBinView builder). This is kludgy and error-prone.
An alternative option would be to require or encourage all buffers allocated by the project to be aligned to some value (e.g., the platform-specific arrow_buffer::alloc::ALIGNMENT, or the max of that and 64/128 bytes, etc).
If we require all of our buffers to be aligned, then that will imply copying unaligned Vecs passed by users (arrow-rs doesn't copy), so we should probably make it obvious & less ergonomic to do so.
Current plan:
- [x] VortexReadAt should not assume alignment of the result bytes.
- #1713
- [X] MessageReader copies to enforce alignment as necessary. The reader is configurable with minimum alignment (default to 64 bytes), and we should also specify minimum alignment in the Buffer flatbuffer. This means we can support VarBin's buffers having 1 byte alignment and never copying, and FastLanes buffers requiring 128 byte alignment.
- #1692
- [ ] File/MessageWriter can be configured to include padding or not. With padding included, the read may have slightly fewer copies (though the reader will always check min alignment). This is similar concept to the writer choosing compression codecs that are faster/slower to decode. Writer's choice. Always. Note we also need to be able to pass the underlying file / stream position to the encoder so it can be globally aligned, not just locally.
- Moved to #1676
- [x] Refactor vortex-buffer to support runtime alignment
- #1742
- [x] ~Introduce an ArrayBuffer struct that wraps
ArrayBuffer{ buffer: Buffer, min_align: usize }so we can capture this information in an ArrayData~. This information is captured in vortex-buffer now.- #1742
- [x] Add a new ScalarBuffer<PType> for passing around typed buffers.
- #1742
- [x] Move IoBuf trait into vortex-io
- #1714
