static-vectors icon indicating copy to clipboard operation
static-vectors copied to clipboard

Aligned Data Vectors

Open Shinmera opened this issue 1 year ago • 2 comments

This is similar to #28, though my specific use-case is to have the payload of a vector be aligned to the page boundary.

This is necessary in order to be able to share data with other processes via posix shm or mmap. I do not have control over the software I'm sharing with, so I can't tell it to start at a different offset. Similarly, because I want to transparently expose lisp vectors to the users of the library, I do not want to allocate a bigger vector and only start filling data in at an offset to that, nor do I want to use displaced arrays, as they are significantly slower than accessing a simple-array.

I talked to Doug (not sure if he has a pingable github account?) about doing this at SBCL'25 and he seemed to think that it should be easy to add. He even started on an implementation, though he didn't finish it. He told me to take up contact with you to discuss this issue, so here I am.

Mallocing a bigger region and punting the vector header and size before the page boundary is easy enough to do, but I suppose we need some sort of additional support in order to be able to pass the correct starting address when freeing the vector again, which I'm not sure how to facilitate best.

We also didn't look at any other implementations than SBCL, so not sure how this would have to be handled there.

Shinmera avatar May 13 '24 16:05 Shinmera

@Shinmera because the underlying allocator cannot guarantee both page alignment and enough space for a header, the implementation will inevitably have to allocate more space (padding the beginning of the array with exactly the alignment size).

And because I don't want to only make this available on SBCL, I'd be inclined to expose a primitive that returns exactly what you said you don't want: a vector and an offset that is aligned on the requested boundary. I don't see how we can make this useable anywhere else.

sionescu avatar May 13 '24 21:05 sionescu

I understand that we have to malloc up to one full extra page that will be wasted, and I am fine with that, that's not a concern. What I care about is that I can view the same data block in exactly the same way on both lisp side and foreign side.

Shinmera avatar May 13 '24 21:05 Shinmera

@Shinmera I'll come up with something shortly.

sionescu avatar May 14 '24 16:05 sionescu

@Shinmera Take a look at https://github.com/sionescu/static-vectors/pull/39

sionescu avatar May 28 '24 00:05 sionescu

Code merged.

sionescu avatar May 29 '24 18:05 sionescu