aligned icon indicating copy to clipboard operation
aligned copied to clipboard

Add features to integrate with generic_array crate

Open cbeck88 opened this issue 3 years ago • 1 comments

In a project, I have to manipulate large aligned arrays of bytes and manipulate them in various ways very quickly. Right now the only way to really do that in generic code in rust is to use generic_array AFAIK, and const generics are still a long way off. Initially the thing I wanted was to make generic_array::Slice API work without losing alignment information. There was no way to do that without adding code to the aligned crate due to the orphan rules, so that was what I did. I found that it worked really well for me.

It's possible that this code should not be in aligned crate, and instead it should be in generic_array crate, which should depend on aligned. But at present that's unworkable because aligned depends on as_slice, and as_slice depends on generic_array. So if generic_array depends on aligned we get a circular dependency. We could resolve this by first making generic_array depend on as_slice rather than the reverse, but that's a larger project.

Later, I added more helpful functionality extending the aligned + generic_array integration idea:

  • AsAlignedChunks allows to iterate over an aligned array in a manner similar to chunks API on the slice type, but preseving the alignment information.
  • AsNeSlice allows to convert a ref to aligned bytes to a ref aligned integer types like u32 and u64. Without alignment information this isn't possible without copies of course, but when we have alignment info we can convert to native-endian integers without copying. It's not uncommon to have to do this in low-level code and I found having a simple, safe API for this helpful.

Both of these were really helpful for my project, because it means that the crates using these calls can deny unsafe code, when otherwise they would have dozens of unsafe calls.

There was also some other minor stuff:

  • Aligned<A4, _> implements AsRef<Aligned<A2, _>> and so on. To do this, I added an associated type to sealed::Alignment type, which is the typenum corresponding to the alignment value. This doesn't become part of the public API and makes it straightforward to implement the trait bounds for these APIs. typenum is in many ways annoying, but I don't know any real alternative, and it is the basis of generic_array anyways.
  • I added A32 and A64 type to this crate. The reason I did that is that some AVX2 instructions require 32-byte alignment and some AVX512 instructions require 64-byte alignment and I need to use them.
  • When I looked at the diff, I realized I added repr(C) to the A types, I think I did that because I thought there might be some rule that everything in the struct is transitively repr(C). But now I think I might be wrong about that, or at least I can't find evidence. But anyways it seems like it can't hurt. Let me know what you think.
  • I marked many functions #[inline], because in my understanding, rust is not permitted to inline across crate boundaries unless that annotation is there, and many of these functions should ideally be completely transparent to the optimizer. Let me know what you think about that.

So, this is somewhat a significant extension of the crate. I think it's good and it has helped me a lot, I've iterated on this API for a few months, and I haven't changed any aspect in a month. And I thought pretty carefully about making an API that will feel familiar and be infallible and not cause overheads.

I'm not sure if you'll agree with the direction here -- let me know what you think. I'd love to find a way to get this code into either aligned or generic_array or split among both crates, because I think much of it needs to be in one of those two places due to orphan rules.

I think it's pretty cool that both size and alignment information can be tracked at compile time this way in rust, and actually be checked by the type system.

There are also some TODO's that I marked, but you should give me direction there what you think is best if you're interested in this patch. Thanks

cbeck88 avatar Sep 24 '20 03:09 cbeck88

I have ended up publishing my fork as aligned-array, thanks again for providing a starting point: https://crates.io/crates/aligned-array

cbeck88 avatar Feb 14 '21 19:02 cbeck88