Unsound implementation of `transmute_vec_as_bytes` in `fyrox-core`

[shinmao]

Hi, we are researchers from SunLab. We consider the safe function transmute_vec_as_bytes includes the unsound implementation.

transmute_vec_as_bytes

https://github.com/FyroxEngine/Fyrox/blob/3b03ea717d201be24dac98cc9e8ebf812808e898/fyrox-core/src/lib.rs#L325-L331 When casting type to u8 slice, we also need to make sure whether the type won't contain padding bytes. Otherwise, it could lead to uninitialized memory access or break the reliability of program. Based on the usages of the function, we consider the generic type T should also implement Pod trait.

PoC

use fyrox_core::transmute_vec_as_bytes;

#[derive(Copy, Clone)]
struct Pad {
    a: u8,
    b: u32,
    c: u8
}

fn main() {
    let pd = Pad { a: 0x1, b: 0x2, c: 0x3 };
    let mut v = Vec::new();
    v.push(pd);
    let fv = transmute_vec_as_bytes(v);
    println!("{:?}", fv);
}

In the program above, we passed the struct that might contain padding bytes as the generic type T. First, when we run with miri, we can get the following results:

error: Undefined Behavior: using uninitialized data, but this operation requires initialized memory
   --> /home/rafael/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:471:5
    |
471 | /     impl_Display!(
472 | |         i8, u8, i16, u16, i32, u32, i64, u64, usize, isize
473 | |             as u64 via to_u64 named fmt_u64
474 | |     );
    | |_____^ using uninitialized data, but this operation requires initialized memory

In the library, we found that this function uses Vec as input as internal usages, which will not trigger the bug. However, since this function is provided as public safe API, we should consider the constraints on the generic type T. (It is required also because the function name is transmute_"vec"_as_bytes :)

The consequence of UB

We could find that the results of fv break the reliability of program under different environment. Compiled with x86_64, the results would be:

[2, 0, 0, 0, 1, 3, 0, 0]

While compiled with x86, the results would be:

[2, 0, 0, 0, 1, 3, 233, 247]

Take the following usages for example, https://github.com/FyroxEngine/Fyrox/blob/25a229690ccb38ebd90bc9e6f81d4bc90b4e752e/fyrox-impl/src/scene/terrain/mod.rs#L101-L112 The Texture built on this bytes could have different results...