Add `core::mem::offset_of!` RFC

[thomcc]

I started this a jillion years ago, and have been meaning to clean it up and submit it for a long time. The intention is to remove most (ideally all) need for hardcoding offsets, computing offsets manually, or using {memoffset,bytemuck}::offset_of!.

CC @JakobDegen, @saethlin (since they provided some amount of input on this during RustConf)

Rendered

[thomcc]

Oh right, CC @rust-lang/wg-const-eval since this RFC allow using this macro during const-eval.

[BurntSushi]

I am personally very much in favor of something like this.

I do wonder if the lang team should be roped into this? From looking at bytemuck's implementation, it seems clear to me that this is something that can be implemented in library code. But, invoking it with a type as an "argument" is also something that feels outside of libs-api and more in lang territory. That is, the way this macro is used, it looks like a new language verb that's defined as a macro in order to get around adding a new language verb. (I don't mean that as any sort of accusation of subterfuge!)

And maybe that's all okay. I don't know. I don't think I mind having this as a macro. But that it might be something the lang team wants to evaluate too. (And they might just say, "nah libs-api can deal with this." But they might not.)

[Lokathor]

As a user, I'd prefer if the docs could say "this info isn't some goofy library calculation, this info is directly from the compiler's data structures, guaranteed accurate, guaranteed no UB, guaranteed no runtime cost."

Which is to say, yeah, "language verb" is probably a better framing.

[afetisov]

One important thing is missing from the reference: the layout of #[repr(rust)] data is unspecified, but is it even true that the layout is fixed? I can totally imagine a compiler optimization that e.g. uses different layout for different local variables of the same type, provided they do not "escape" in some sense (e.g. a type T is private and no explicit pointers to T are created). Without such guarantees any attempts to manipulate individual fields via offsets are meaningless (at least unless the offset is used for accessing the same "instantiation" of T, but that's generally impossible to enforce).

For example, the reference explicitly states that the layout of #[repr(rust)] types can vary between different compilations. This much more tame than the possibility above (vary with each use of type), but may also lead to subtle errors.

I think the RFC should have some motivation for allowing offset_of! on non-#[repr(C)] types. "memoffset does it" isn't sufficient justification, since memoffset has no way to enforce the layout requirements on the supplied types.

---

Being a macro, offset_of! also can't distinguish between structs and enums, or between fields and arbitrary garbage. I'm afraid this will lead to some very poor post-monomorphization errors. If offset_of! calls are embedded in other macros, the errors can be super cryptic.

---

One possibility which isn't considered in the RFC is to use special constants instead of an expression macro. Now, defining those constants manually is mentioned, and has many problems, but what if they are defined by a compiler macro? For example, there could be some #[derive(FieldOffsets)] derive macro which would provide the constants with the values substituted by the compiler, e.g.

// before expansion
#[derive(FieldOffsets)
struct Foo {
    first: u32,
    pub second: Vec<Bar>,
    pub(crate) third: (u8, bool, usize),
}

//after expansion
impl Foo {
    const FOO_FIRST_OFFSET: usize = 0;
    pub const FOO_SECOND_OFFSET: usize = 8; // for example
    pub(crate) const FOO_THIRD_OFFSET: usize = 32; // for example
}

For better ergonomics, the constants may be accessible via some expression macro offset_of!, which would expand to the relevant constant name.

• A derive macro trivially sidesteps the issues of const evaluation and lack of runtime overhead (since the values of constants are inlined by the compiler).
• It easily deals with unsupported types of data (enums etc), and can more easily support them in the future (since it has full information about the type definition).
• It automatically mirrors the privacy rules for type and fields.
• It can transparently handle any other requirements on the type, e.g a specific #[repr] (just check for desired attributes).
• It could even provide more granular control to the representation of the type: if the author doesn't provide the #[derive(FieldOffsets)], then the consumer code cannot rely on the specific offsets.
• This information can also be more granularly supported for specific fields via derive macro attributes (like #[offset_of(ignore)] or #[offset_of(pub(crate))] pub field: T.

This implementation is also very hard to provide in a library crate, which provides a stronger argument for including it into the core.

[mikeleany]

I've always wanted this feature, but I'm wondering what the rational would be for not supporting ?Sized fields from the beginning.

[thomcc]

@BurntSushi:

I do wonder if the lang team should be roped into this

Yeah, I didn't know what to tag this as. I'm certainly not opposed to roping in t-lang, and added the label.

---

@Lokathor:

this info isn't some goofy library calculation, this info is directly from the compiler's data structures, guaranteed accurate, guaranteed no UB, guaranteed no runtime cost

The intention is that's how this would be implemented (that's what the bit at the end of the reference is intended to mean), but "no runtime cost" is kind of difficult to define in a way that still works in implementations that don't make as firm of a distinction between compile time and runtime.

Anyway, this felt likely to bog down the implementation under bikeshedding as to what this means, and in practice "evaluates to a constant expression" should be sufficient.

I also don't want to forbid a pure-library implementation, although I believe implementing it as a builtin macro is likely to be better for several reasons (error messages, compile-time performance, reliability, ...).

---

@mikeleany:

I've always wanted this feature, but I'm wondering what the rational would be for not supporting ?Sized fields from the beginning.

Hm, largely because it prevents a pure library implementation. That said, I don't really care about that, and perhaps with enough cleverness it would still be possible.

[thomcc]

@afetisov (your comments are long enough to get a separate reply)

I think the RFC should have some motivation for allowing offset_of! on non-#[repr(C)] types. "memoffset does it" isn't sufficient justification, since memoffset has no way to enforce the layout requirements on the supplied types.

The use cases for this are largely the same as the use cases for using offset_of! on repr(C) types -- {de,}serialization, debugging support code, FFI, and data structures -- in all of these it may be undesirable or impractical to force all types to use repr(C).

More generally, it is a goal of this RFC to remove the reasons people have to implement offset_of! manually (either explicitly, or often in an ad-hoc manner). If core::mem::offset_of! lacks functionality that you could implement yourself, then this adds more motivation to not use it.

One important thing is missing from the reference: the layout of #[repr(rust)] data is unspecified, but is it even true that the layout is fixed?

In several ways it is already fixed at compile time (size and alignment are, which limits the size of the overall structure). In other ways, it is fixed at runtime. Allowing use in const evaluation prevents an implementation from doing things like randomizing field offsets at startup, unless it also performs modifications to adjust offset_of! values (and calculations derived from them).

My suspicion is that many implementations which can randomize field offsets at startup can recompute constant values (because they're likely interpreted), but it's plausible that there's some middle-ground.

Either way, this is more or less what I'm getting at by the first item under "Drawbacks" -- previously this information was only available at runtime, and this allows accessing it at compile time.

I can totally imagine a compiler optimization that e.g. uses different layout for different local variables of the same type, provided they do not "escape" in some sense (e.g. a type T is private and no explicit pointers to T are created)

This kind of optimization (and more aggressive ones, like SROA) would still be allowed in exactly the same cases they are currently allowed. If a pointer to an instance of a structure does not escape, the offsets of its field do not matter. This is true regardless of the #[repr] of the type, and is not hindered or changed by offset_of! at all, as that information can already be computed using address arithmetic on pointers to the structure and field.

Being a macro, offset_of! also can't distinguish between structs and enums, or between fields and arbitrary garbage. I'm afraid this will lead to some very poor post-monomorphization errors

Are you sure? From my experience with the code in rustc_builtin_macros and from conversations with compiler contributors (which I had when preparing this RFC), none of this should be a problem.

Perhaps someone familiar with compiler internals cares to weigh in?

One possibility which isn't considered in the RFC is to use special constants instead of an expression macro

I'll add it to the RFC as an alternative, but I am not a fan of macros that expand to items not present in the source (you'll note that none of the existing builtin #[derive()]s do this); I find it results in poor tooling experience, and is generally confusing. Additionally:

<a derive macro> can more easily support them in the future (since it has full information about the type definition).

I don't think there's any reason a derive macro would have more information here than is available to a builtin macro. They're essentially the same thing, and have essentially the same limitations.

This implementation is also very hard to provide in a library crate, which provides a stronger argument for including it into the core.

I don't think this is true -- if core::mem::offset_of! is added, I think it would be quite straightforward to implement a #[derive(FieldOffsets)] equivalent to what you describe as a procedural macro library... What would be difficult about it?

---

I'll try to update drawbacks/alternatives/motivation with this feedback.

[JakobDegen]

is it even true that the layout is fixed? I can totally imagine a compiler optimization that e.g. uses different layout for different local variables of the same type, provided they do not "escape" in some sense

Yes, the layouts are fixed. However, the as-if rule applies anyway. If the compiler can prove that the user can't tell the difference either way, it's always free to do whatever it wants as long as the behavior of the program is correct.

Being a macro, offset_of! also can't distinguish between structs and enums, or between fields and arbitrary garbage. I'm afraid this will lead to some very poor post-monomorphization errors. If offset_of! calls are embedded in other macros, the errors can be super cryptic.

offset_of! is less a macro and more a compiler built-in. The compiler certainly can distinguish between structs and enums perfectly well, it does this already when it allows you to write things like MyStruct { foo: 5 }. This macro would use much of the same logic. There also means that we would reject things like fn foo<T>() { offset_of!(T, field); } and so there is no possibility of post-mono errors.

[afetisov]

If a pointer to an instance of a structure does not escape, the offsets of its field do not matter.

The pointer may not be computed explicitly, but perhaps there is still some roundabout way to get it via unsafe code. Trying to bound the behaviour of unsafe code hits Rice's theorem fast, and I can't imagine what would a reasonable definition of UB for offset_of! look like. "It is UB to access fields via this offset without explicitly using *mut T at some point"? That doesn't sound sensible.

Unless the reference explicitly guarantees that a type is always represented in the same way in memory, I would feel very uneasy about exposing offset_of! for unspecified layouts. It's also a semver footgun, but that's already mentioned in RFC.

that information can already be computed using address arithmetic on pointers to the structure and field.

If the layout stability within a single artifact is not guaranteed, then all such calculations are borderline UB (their results can not be used for any unsafe operations in any nontrivial case).

Yes, the layouts are fixed.

Thank you, but that doesn't carry much weight unless it passes an RFC and is specified in the Reference. For all I know, that's just the current implementation restriction.

The use cases for this are largely the same as the use cases for using offset_of! on repr(C) types -- {de,}serialization, debugging support code, FFI, and data structures -- in all of these it may be undesirable or impractical to force all types to use repr(C).

I would like to see specific use cases. I can't imagine how field offsets would help with (de)serialization of types with unspecified layouts, you certainly can't just transmute a slice of incoming data. Nor can I imagine how layout-unstable types could possibly be used in FFI, except as opaque types with no access to inner fields.

What would be difficult about it?

Well, that depends on the implementation and power of offset_of!. For example, it may use something which make the call UB in some cases, or it may be unusable in const contexts at the moment. But I guess being a builtin macro could grant it the necessary special powers. I don't know how complex a builtin macro can be.

[Lokathor]

I would like to just put in the standard reminder that the Reference is a best effort documentation of how the compiler/language currently works, it's not actually an authority. If the reference and the compiler conflict about a point, it's just as likely that the reference will be updated as the compiler will be.

[JakobDegen]

The pointer may not be computed explicitly, but perhaps there is still some roundabout way to get it via unsafe code. Trying to bound the behaviour of unsafe code hits Rice's theorem fast

Fortunately, the language doesn't have to worry about this. People writing compiler optimizations will be responsible for proving that the results of the optimization are not observable.

Thank you, but that doesn't carry much weight unless it passes an RFC and is specified in the Reference.

I don't really know how to respond to this besides saying that this is what it means for a type to have a layout. If the layout is not fixed, then it is not a property of the type, but a property of some other thing

[Lokathor]

Nor can I imagine how layout-unstable types could possibly be used in FFI, except as opaque types with no access to inner fields.

I got one: With the GPU you often have to send structured data and the way you do it is basically sending a bunch of bytes along with a description of what the fields of each field within each struct is. Details vary by API so I don't want to get into it too heavily, but you'd tell the API something like "the 0th field is an [f32;2] at offset 0, the 1th field is an [f32;3] at offset 8" or whatever, and then the GPU would read your buffer that way.

[afetisov]

I don't really know how to respond to this besides saying that this is what it means for a type to have a layout.

I guess my question can be rephrased as "is it guaranteed that #[repr(rust)] types have a layout?" The reference says that no guarantees on the default layout are provided, and no information besides size and alignment are explicitly exposed. "No guarantees on layout" may well be interpreted as "no specific layout at all".

[Kixiron]

For something to have an unspecified layout it by definition must have a layout

[CryZe]

For something to have an unspecified layout it by definition must have a layout

Well of course it has a layout. But the question is if there might be situations in the future where the compiler might want to optimize the layout based on the particular case where the variable is used, just like how a struct might never be on the stack or heap at all and instead be represented entirely in registers. Similar optimizations could be possible where the struct is sorted in a particular way in one function and differently in another. Stabilizing this macro for repr(Rust) would prohibit such optimizations.

[carbotaniuman]

This macro is already writable with stable Rust, with 2 library implementations of varying QoL. A compiler that wishes to reorder structs (in a way as described above) must already ensure that such reordering does not impact the current library implementations. (This RFC, as written, allows a library-only implementation, even if that might not be the preferred route).

[CryZe]

You certainly can write the macro in stable Rust, but you still need unsafe to actually use that offset. So there's still a possibility that this offset is actually unsound to use then on a repr(Rust) type. However it's possibly indeed too late to allow such an optimization as people are probably relying on it enough already to break that.

[Diggsey]

You certainly can write the macro in stable Rust, but you still need unsafe to actually use that offset. So there's still a possibility that this offset is actually unsound to use then on a repr(Rust) type. However it's possibly indeed too late to allow such an optimization as people are probably relying on it enough already to break that.

I've had a crate which wraps the concept of a field offset in a safe API which has existed for 6 years 😛

https://docs.rs/field-offset/latest/field_offset/index.html

[RalfJung]

I've had a crate which wraps the concept of a field offset in a safe API which has existed for 6 years

But that does not mean that the crate is sound. ;)

I agree that in theory, repr(Rust) could be some wild dynamic thing where the concept of a "field offset" does not apply, and where unsafe code trying to compute and then use such an offset would be unsound. And I don't think we have ever explicitly stated that repr(Rust) does not do funny stuff like this. But I doubt anyone actually wants to allow funny stuff like this...

Note however that Foo<T> and Foo<U> are allowed to differ arbitrarily in their layout, even if the type parameter is used only inside PhantomData. This caveat should be documented very clearly in the offset_of docs.

The use cases for this are largely the same as the use cases for using offset_of! on repr(C) types -- {de,}serialization, debugging support code, FFI, and data structures -- in all of these it may be undesirable or impractical to force all types to use repr(C).

repr(Rust) is invalid for many of these since even just recompiling the program unchanged can lead to different layouts.

Basically, it would be totally correct and following the spec to enable -Zrandomize-layout by default.

I've always wanted this feature, but I'm wondering what the rational would be for not supporting ?Sized fields from the beginning.

The unsized field of an unsized struct/tuple does not have a fixed layout, so offset_of makes no sense for such a field (not as a compile-time constant, anyway). But we could allow it for the other fields, yeah.

[programmerjake]

The unsized field of an unsized struct/tuple does not have a fixed layout, so offset_of makes no sense for such a field (not as a compile-time constant, anyway). But we could allow it for the other fields, yeah.

The unsized field may not have a fixed layout, but it often can have a fixed offset, thus offset_of! makes sense to have. e.g.:

#[repr(C)]
struct S {
    a: i32,
    b: [u16],
}

here S::b always has offset 4, even though its size may vary.

[thomcc]

repr(Rust) is invalid for many of these since even just recompiling the program unchanged can lead to different layouts.

It's invalid if you can't access field offset information. If you can a lot of these become significantly more tractable -- For example the FFI use case described by @Lokathor. For serialization, consider formats that describe what offset is used for each field of the record in a payload, etc.

In other worse, you cannot assume repr(Rust) has a consistent layout, but using offset_of! you can get information that describes the layout in sufficient detail for code elsewhere to reconstruct the data.

These use cases (if correctly implemented) would work even if -Zrandomize-layout is enabled by default, and people do use existing offset_of implementations for them.

[RalfJung]

The unsized field may not have a fixed layout, but it often can have a fixed offset, thus offset_of! makes sense to have. e.g.:

Often but not always, e.g. in (i32, dyn Send), the offset of the 2nd field is not fixed.

[RalfJung]

For serialization, consider formats that describe what offset is used for each field of the record in a payload, etc.

This assumes that everything we need to know about struct layout can be described by size, alignment, and field offsets.

I agree that should be the case, but if that is an intended usecase of the RFC, it should be stated explicitly.

[programmerjake]

The unsized field may not have a fixed layout, but it often can have a fixed offset, thus offset_of! makes sense to have. e.g.:

Often but not always, e.g. in (i32, dyn Send), the offset of the 2nd field is not fixed.

since the offset only depends (or can be made to only depend) on the other Sized fields and the alignment of the ?Sized field, it should be fine to use offset_of! on any field with type T: ?Sized + KnownAlignment where the alignment is known at monomorphization time. All slices and str would implement KnownAlignment because their alignment is the alignment of the element type, which is Sized. dyn types would have to be dyn MyTrait + KnownAlignment to be usable with offset_of!.

[programmerjake]

perhaps Aligned is a better name than KnownAlignment. for backward compatibility, Sized: Aligned and T: ?Sized implies T: ?Sized + ?Aligned

[thomcc]

Something like KnownAlignment/Aligned is overkill -- align_of_val has the exact same issue here (not really needed for slices and types with trailing slices) and we get by without something like that. If we want to add that, it would be a totally separate thing. Anyway, I think the consistent fix for that would be an offset_of_val!, analogous to size_of_val or align_of_val.

That said, I suspect offset_of_val! is niche enough that it should be deferred to future work (I'll add it to that section when I update the RFC, hopefully later today).

[RalfJung]

dyn types would have to be dyn MyTrait + KnownAlignment to be usable with offset_of!.

That would not work -- dyn MyTrait + KnownAlignment implies that the dynamically chosen type has known alignment, but we still don't know that alignment from the type alone. (Dynamically we already know the alignment, via align_of_val.)

@thomcc I feel like it makes sense to support asking for offsets of all the sized fields on an unsized type -- if we want to commit to those always being statically known.

[programmerjake]

dyn types would have to be dyn MyTrait + KnownAlignment to be usable with offset_of!.

That would not work -- dyn MyTrait + KnownAlignment implies that the dynamically chosen type has known alignment, but we still don't know that alignment from the type alone. (Dynamically we already know the alignment, via align_of_val.)

ah, yeah. dyn Trait + Aligned<ALIGNMENT = N> would work though, for const generic N (assuming const generic stuff is implemented enough yet).

[afetisov]

A couple extra objections against exposing offsets for #[repr(Rust)] types (and any other types with underdefined layout).

• Although the layout of #[repr(Rust)] types is undefined, exposing offsets means that the layout is now an observable behaviour. This means that it would de facto become fixed and relied upon by end-users, and could no longer be changed in the future. This runs counter to the goal of no ABI stability for default layout, since it would prevent further layout optimizations.

• In particular, currently the layout is not guaranteed to be stable even between different compiler runs of the same compiler on the same codebase. But that means that exposing offsets for default layout make possible horrible nonreproducible bugs: the runtime behaviour may branch on the values of offsets, thus successively compiled binaries will behave in entirely different and non-reproducible way. Paired with optimizations, that's as close as it gets to true Undefined Behaviour, which just shouldn't be possible in safe Rust.

• As a consequence of the above, if offset_of! is const, then the crate may fail to compile on different compiler invocations (it's easy to write either compile-time assertions or types which fail to be well-formed based on the values of these constants). Nondeterministic compilation errors are unprecedented in current Rust and most if not all programming languages. They also make any good diagnostics impossible.

• Let's say the compiler randomizes the layouts based on some randomness, which can be provided as explicit seed during the compilation. Although an obscure hack, this will eliminate the nondeterministic errors above. However, this would make the compiler behaviour unreproducible by any tool other than the specific compiler version itself. For example, IDEs need to be able to do constant evaluation, for their diagnostics and refactorings. However, since the layout algorithm is an internal, potentially very complex compiler detail which may change between versions, it cannot be reproduced in the IDE const evaluation engine. This will make any non-rustc tools fundamentally and unrepairably broken on codebases which do complex offset_of! tricks. Since those tricks can be wrapped in crates and easily distributed, this will likely make them broken on any sufficiently complex real-world codebase.

TL;DR: exposing offsets will effectively ossify the layouts of types, thus it must not happen on types with not fully defined layouts. Extending offset_of! to new types and layouts must be preceded by an RFC which fully stabilizes the corresponding layout.

[JakobDegen]

@afetisov none of those things don't apply just as much to size_of. Is your claim that providing that was a mistake too?

exposing offsets means that the layout is now an observable behaviour

This is already the case. memoffset and such crates already exist today. You can cast pointers to integers to observe the offset of various fields.

But that means that exposing offsets for default layout make possible horrible nonreproducible bugs

This is not really the case in practice; the compiler does not gratuitously change layouts just because it can. That's not to say that this doesn't happen at all of course, but executing the same build twice won't just randomly change things. (and in any case, there is still a requirement for reproducible builds in general)

as a consequence of the above, if offset_of! is const, then the crate may fail to compile on different compiler invocations

This is already the case today with size_of. There are many crates (including lots of rustc_* crates) that const_assert!(size_of::<SomeType>() == 24) or whatever. We have been willing to break those in the past, and there is no reason that that would change.