zerocopy

zerocopy copied to clipboard

This issue tracks work which will need to land before ring can take a dependency on zerocopy. See this comment for context.

I put up a prototype: https://github.com/briansmith/ring/pull/1693

• [x] https://github.com/google/zerocopy/pull/434
  • This is needed to support the BitXorAssign trait, currently implemented here
• [x] https://github.com/google/zerocopy/pull/183
  • transmute_ref! is needed to support reference transmutations between array types such as &[U32<BigEndian>; 4] -> &[u8; 16], which are currently supported in ring with the ArrayEncoding::as_byte_array method
• [ ] Determine whether zerocopy meet's ring's bar for performance and constant time guarantees (briansmith/ring#1693 (comment))
• [x] https://github.com/google/zerocopy/issues/438
  • Note that this is a breaking change; as of this writing, the most recent published version is 0.7.6, so I expect this will be released in 0.8.0

Previously, we tracked #254, but this comment implies we don't need it.

cc @briansmith

Add aligned byteorder types #254

• This might be a blocker if the fact that byteorder types currently have no alignment requirement results in a performance regression

In ring, usually (ideally always) we already have a wrapper type like Nonce or Counter where we could do the alignment if we really need it. Ultimately we want everything to be aligned so that we can do 128-bit or 256-bit or 512-bit vector operations on the bytes, without any performance penalty, even on small chips like RISC-V. I don't think that aligning to 4 bytes for u32s or 8 bytes for u64s gets us to there.

Would it be practical to factor out the safe transmutation aspect of zerocopy from the rest of it? My understanding is that zerocopy is trying to do many things at once: (a) implementing safe transmutation by modeling what transmutations are safe within the type system, (b) zero-copy parsing of unaligned multi-byte data structures, (c) endian conversion and convenience methods on top of that. The recent endianness work was at least partly motivated by ring but what ring really needs is really (a). We don't do (b) at all and it looks like we'll have to do (c) ourselves eventually as zerocopy (reasonably) won't make the free-from-compiler-introduced-side-channel guarantees that we're aiming for. In some sense (c) is actually kind of a hazard that we'd need to guard against in ring.

(For webpki (X.509 certificate processing) we don't need any of this as X.509 uses ASN.1 which doesn't use fixed-length data structures that could be modeled by (b). For TLS (a) isn't needed but potentially (b) is useful at the record layer, which does use fixed-length structures very much like TCP/IP.)

Would it be practical to factor out the safe transmutation aspect of zerocopy from the rest of it? My understanding is that zerocopy is trying to do many things at once: (a) implementing safe transmutation by modeling what transmutations are safe within the type system, (b) zero-copy parsing of unaligned multi-byte data structures, (c) endian conversion and convenience methods on top of that. The recent endianness work was at least partly motivated by ring but what ring really needs is really (a). We don't do (b) at all and it looks like we'll have to do (c) ourselves eventually as zerocopy (reasonably) won't make the free-from-compiler-introduced-side-channel guarantees that we're aiming for. In some sense (c) is actually kind of a hazard that we'd need to guard against in ring.

(For webpki (X.509 certificate processing) we don't need any of this as X.509 uses ASN.1 which doesn't use fixed-length data structures that could be modeled by (b). For TLS (a) isn't needed but potentially (b) is useful at the record layer, which does use fixed-length structures very much like TCP/IP.)

Yeah I think this is a fairly accurate taxonomy of what zerocopy provides. The way it breaks down in my head is similar:

• Analysis of types (ie, the custom derives provided by zerocopy-derive)
• Various unsafe utilities that build on top of this analysis (the various methods in our unsafe traits, the Ref type, etc)
• Nice-to-have utilities which don't require any unsafe

The former two buckets definitely belong in zerocopy, but the latter (which includes the byteorder module) might not.

For your purposes, perhaps you could fork the byteorder module? The advantage of having it in zerocopy is that you can use it without enabling the derive feature, and thus keep your crate dependencies small. If you were to fork it, you'd have to depend on zerocopy with the derive feature enabled. But if it's important to you to have your dependency on zerocopy be minimal in terms of how much heavy lifting it's doing for you, that'd be a viable approach.

The byteorder module is a bit convoluted because of its use of macros, but roughly speaking the core just looks like this:

#[derive(FromZeroes, FromBytes, AsBytes, Unaligned)]
#[repr(transparent)]
pub struct U32<O>([u8; 4], PhantomData<O>);

That's obviously something you could just do in ring (either be fully forking byteorder or just be reimplementing the bits you need by hand).

With recent refactorings to ring (one currently in progress) we should reevaluate what, if anything, needs to be added to zerocopy after we found that the endian stuff in ring could mostly be deleted. I think we may need to do more experiments in ring to see how we can statically guarantee panic-free-ness and safety with the help of zerocopy.

I think we may need to do more experiments in ring to see how we can statically guarantee panic-free-ness and safety with the help of zerocopy.

Can you say a bit more about which code paths you're referring to here? We've been exploring panic freedom, so I'd be curious to learn about new use cases.

EDIT: I just saw https://github.com/briansmith/ring/pull/1736; I assume you're (at least partly) referring to that?