aligned
aligned copied to clipboard
Add features to integrate with generic_array crate
In a project, I have to manipulate large aligned arrays of bytes
and manipulate them in various ways very quickly. Right now the only way
to really do that in generic code in rust is to use generic_array
AFAIK,
and const generics are still a long way off. Initially the thing I wanted was
to make generic_array::Slice
API work without losing alignment information.
There was no way to do that without adding code to the aligned
crate
due to the orphan rules, so that was what I did. I found that it worked
really well for me.
It's possible that this code should not be in aligned
crate, and
instead it should be in generic_array
crate, which should depend
on aligned
. But at present that's unworkable because aligned
depends on as_slice
, and as_slice
depends on generic_array
.
So if generic_array
depends on aligned
we get a circular dependency.
We could resolve this by first making generic_array
depend on as_slice
rather than the reverse, but that's a larger project.
Later, I added more helpful functionality extending the
aligned
+ generic_array
integration idea:
-
AsAlignedChunks
allows to iterate over an aligned array in a manner similar tochunks
API on the slice type, but preseving the alignment information. -
AsNeSlice
allows to convert a ref to aligned bytes to a ref aligned integer types likeu32
andu64
. Without alignment information this isn't possible without copies of course, but when we have alignment info we can convert to native-endian integers without copying. It's not uncommon to have to do this in low-level code and I found having a simple, safe API for this helpful.
Both of these were really helpful for my project, because it means that the crates using these calls can deny unsafe code, when otherwise they would have dozens of unsafe calls.
There was also some other minor stuff:
-
Aligned<A4, _>
implementsAsRef<Aligned<A2, _>>
and so on. To do this, I added an associated type tosealed::Alignment
type, which is thetypenum
corresponding to the alignment value. This doesn't become part of the public API and makes it straightforward to implement the trait bounds for these APIs.typenum
is in many ways annoying, but I don't know any real alternative, and it is the basis ofgeneric_array
anyways. - I added
A32
andA64
type to this crate. The reason I did that is that some AVX2 instructions require 32-byte alignment and some AVX512 instructions require 64-byte alignment and I need to use them. - When I looked at the diff, I realized I added
repr(C)
to theA
types, I think I did that because I thought there might be some rule that everything in the struct is transitivelyrepr(C)
. But now I think I might be wrong about that, or at least I can't find evidence. But anyways it seems like it can't hurt. Let me know what you think. - I marked many functions
#[inline]
, because in my understanding, rust is not permitted to inline across crate boundaries unless that annotation is there, and many of these functions should ideally be completely transparent to the optimizer. Let me know what you think about that.
So, this is somewhat a significant extension of the crate. I think it's good and it has helped me a lot, I've iterated on this API for a few months, and I haven't changed any aspect in a month. And I thought pretty carefully about making an API that will feel familiar and be infallible and not cause overheads.
I'm not sure if you'll agree with the direction here -- let me
know what you think. I'd love to find a way to get this code into
either aligned
or generic_array
or split among both crates,
because I think much of it needs to be in one of those two places
due to orphan rules.
I think it's pretty cool that both size and alignment information can be tracked at compile time this way in rust, and actually be checked by the type system.
There are also some TODO's that I marked, but you should give me direction there what you think is best if you're interested in this patch. Thanks
I have ended up publishing my fork as aligned-array
, thanks again for providing a starting point: https://crates.io/crates/aligned-array