rust-lang
Clarify transmute rules and behavior of `repr(C)`
In particular the nomicon currently says
Transmuting between non-repr(C) types is UB
Which is (a) not correct -- at best it is "unspecified behavior" because you make assumptions about layout -- and (b) there are some non-repr(C) types which you can actually transmute just fine. repr(transparent) is the obvious case; another one is relying on the niche optimization for Option<&T>.
Cc @Centril
Yea -- tho if we say that it is "unspecified behavior" (for the parts which it isn't well defined as in e.g. transparent) we'll need to impress upon people that we are free to change things in a way which may result in UB and so that people should treat it as-if it were UB.
cc @rkruppe
@RalfJung I'd be interested if this could be clarified (particularly with respect to repr(transparent) being acceptable to transmute). As a concrete example:
Is it UB to cast pointers of one repr(Rust) type to another, assuming that the first is repr(transparent) containing the second? e.g. casting from *mut UnsafeCell<MaybeUninit<A>> to *mut A?
I use this technique in my library BBQueue to allow access to a GenericArray buffer. Here is the struct member definition, and where I cast that pointer through the UnsafeCell and MaybeUninit to create a slice. In this case, both are transparent, which seems to be consistent with the nomicon.
Is it UB to cast pointers of one repr(Rust) type to another, assuming that the first is repr(transparent) containing the second? e.g. casting from *mut UnsafeCell<MaybeUninit<A>> to *mut A?
Basically, "yes if the layouts are sufficiently compatible", which is something that the UCG has been working on.
Though note that for UnsafeCell there are some special rules still in place, namely that an &UnsafeCell<T> may only be turned into a writable pointer by calling get. You may not do a transmute. Internally, a transmute is basically what happens, so this is somewhat silly, but this is done to keep our options open wrt other choices for our aliasing model. (The latest Stacked Borrows is fine with the transmute, but an earlier version was not.)
Is it possible to also mention repr(u*) and repr(i*) types here? Transmuting from u8 to repr(u8) shouldn't be UB.
The PR currently makes no statement either way about that and for a first step, I'd prefer to leave it as that. Documenting better what is guaranteed about layouts is a monumental project which is being undertaken by the UCG at https://rust-lang.github.io/unsafe-code-guidelines/layout.html.