Extern types v2

[Skepfyr]

Define types external to Rust and introduce syntax to declare them. This additionally introduces the MetaSized trait to allow these types to be interacted with in a generic context. This supersedes #1861, and is related to the open RFCs #2984 and #3319.

Rendered

[Aloso]

Should MetaSized be a supertrait of Sized?

Yes, I think so, unless there are plans to add types that are Sized but not MetaSized, which makes no sense IMO.

Personally, I think it would be better to leave the meaning of ?Sized unchanged. MetaSized could be a trait that is automatically implied for trait bounds, so these are semantically identical:

fn foo<T>();
fn foo<T: Sized + MetaSized>();

as are those:

fn foo<T: ?Sized>();
fn foo<T: ?Sized + MetaSized>();

In other words, MetaSized would be opt-out rather than opt-in, like Sized is today. And if MetaSized is a super-trait of Sized, then you can opt-out of both traits with ?MetaSized, which is convenient.

Note that this is not a breaking change and requires no edition migration.

[scottmcm]

then you can opt-out of both traits with ?MetaSized

FWIW, lang has historically said "no more ?Trait bounds" like this. That's why there's no ?Move, no ?Unpin, no ?AlignSized , etc.

So I like the edition switch approach here: the conceptual meaning of ?Sized is still essentially the same, and code that just wants to deal in references can continue to use T: ?Sized, with only code that needs to look at the dynamic size of something needing to add the extra bound. (You can follow old docs that say ?Sized, and if you do something that needs the runtime size it'll be an obvious error message saying to add the + MetaSized.)

[clarfonthey]

Since the general consensus on the syntax is... very mixed, it might be worth focusing on MetaSized and it's semantics first, and maybe delegating the exact way to define extern types to another RFC?

Having a stable MetaSized would allow libraries to determine a strategy of handling extern types while we figure out the syntax, and the semantics seem like the largest barrier to getting that right now.

I just don't want to see more progress on refining Sized semantics for DSTs get canned because of syntax bikeshedding.

[Skepfyr]

@Aloso I've added a note to the alternatioves section about ?MetaSized, but as Scott says the lang team have been opposed to these so this RFC is intentionally written to avoid adding it.

[Skepfyr]

@clarfonthey

Since the general consensus on the syntax is... very mixed, it might be worth focusing on MetaSized and it's semantics first, and maybe delegating the exact way to define extern types to another RFC?

That's an option, although I'm tempted to leave everything as is for now and keep that option in my back pocket for if the extern types syntax is the only thing blocking this. Adding MetaSized without any types that can be !MetaSized would be hard to justify.

[scottmcm]

it might be worth focusing on MetaSized and it's semantics first, and maybe delegating the exact way to define extern types to another RFC?

One potential middle ground would be to add a stable Unknown extern type in core, and for now have the syntax be "make a transparent wrapper on that".

I don't know that I'd call that a 90% solution, but it's maybe a 60% solution.

(I'd rather not punt syntax out if possible, but if it's a matter of landing an edition change in time, I'd rather the bounds change without the syntax than nothing.)

[tgross35]

What is the reasoning that optional traits aren't allowed in supertraits? I feel like a builtin trait JustSizedEnough: ?Sized + MetaSized {} may be useful to reduce the clunkiness of ?Sized + MetaSized. This combined bound is expected to be used more often than just ?Sized outside of FFI, right?

Also, is this understanding in this table correct? It might be good to include some kind of table or diagram in the RFC, I've found it kind of hard to wrap my head around just the text.

~~bound~~ bound-like thing	implied by the ~~bound~~ bound-like thing	note
Sized	MetaSized	implemented for everything by default
MetaSized	nothing (?Sized)	unsized types like str or [T] (?Sized + MetaSized is needed to allow these types)
!Sized	!MetaSized	something that is !Sized can never be MetaSized

[CAD97]

This combined bound is expected to be used more often than just ?Sized outside of FFI, right?

An owning pointer use will generally want ?Sized + MetaSized, and a non-owning pointer use will generally want ?Sized + ?MetaSized. The size of a non-owned object is only really needed if you're going to replace it, which happens fairly rarely in generic code, and when it does, usually you are also manipulating values by-ownership.

Interestingly, with implied bounds, fn<T: ?Sized>(Box<T>) would be able to still imply T: MetaSized. I don't know if the fundamentalness of MetaSized are enough to overwhelm the known pitfalls with implied bounds, though.

What is the reasoning that optional traits aren't allowed in supertraits?

Because ?Sized isn't adding a bound, it's opting out of one. When you write fn<T>, there's an invisible implied bound of T: Sized, unless you say that T: ?Sized.

Traits don't have this default implied bound of Sized, so ?Sized doesn't mean anything as a supertrait bound. Writing fn<T: JustSizedEnough> is the same as writing fn<T: Sized + JustSizedEnough>, and writing fn<T: ?Sized + Sized> is the same as writing fn<T: Sized>.

I didn't actually review the RFC for this, but if it doesn't yet, it should say that traits have a default MetaSized bound in editionBefore and no longer do in editionAfter (matching the behavior of ?Sized). It also needs to specify how this interacts with dyn; is Self: MetaSized dyn-safe, is Self: ?MetaSized, and when does dyn Trait: MetaSized hold?

!Sized

!Sized isn't a bound. !Sized is a colloquial way of referring to types where T: Sized doesn't hold, so colloquially we would say that str is !Sized + MetaSized. T: ?Sized thus says to permit both Sized and !Sized types.

[Skepfyr]

@tgross35 I agree with what CAD97's said, I'm confused by your table and don't understand what you're trying to convey. Sized implies MetaSized (in fact I'm now convinced that it should be a supertrait of Sized) so all things that are Sized are MetaSized, but some things are not Sized but are MetaSized (eg trait objects), and finally extern types are neither.

@CAD97

I don't know if the fundamentalness of MetaSized are enough to overwhelm the known pitfalls with implied bounds, though.

I'm beginning to worry that this RFC will require implied bounds to be acceptable, can you link to the known pitfalls with implied bounds?

I didn't actually review the RFC for this, but if it doesn't yet, it should say that traits have a default MetaSized bound in editionBefore and no longer do in editionAfter (matching the behavior of ?Sized). It also needs to specify how this interacts with dyn; is Self: MetaSized dyn-safe, is Self: ?MetaSized, and when does dyn Trait: MetaSized hold?

I hadn't considered this actually, I'll update the RFC soon. dyn Trait is always MetaSized (in fact the vtable directly holds the size and alignment so the implementation is nice and obvious) so I think there isn't much interaction between this and object-safety.

[Skepfyr]

For anyone following along there was a lang team design meeting about this yesterday, the minutes are available here: Extern types V2 - HackMD. There's also some Zulip discussion here: #t-lang > Design meeting 2023-08-09 - rust-lang - Zulip

That meeting came out with some suggestions for investigations that would help move this RFC along, the current set of things I think need doing are:

• [ ] Clarify dereferencing is allowed but place-to-value coercion is not.
• [ ] Remove uses of "opaque types" to reduce confusion with impl Trait
• [ ] Make MetaSized a supertrait of Sized
• [ ] Clarify implicit MetaSized bound on traits that drops on edition change.
• [ ] Investigate how many ?Sized bounds would now need + MetaSized
• [ ] Investigate whether the MetaSized bound can be removed on Box (and similar) so that it no-one needs to type it when just mentioning boxed things (the drop impl currently needs it to deallocate).

[bjorn3]

Sized is an auto trait, but auto traits can't have trait bounds I believe.

[Skepfyr]

Unless I'm missing something that could entirely prevent this kind of thing from working?

fn size_of_val<T: ?Sized + MetaSized>(val: &T) -> usize { ... }
fn foo<T>(val: &T) -> usize {
    size_of_val(val)
}

[tgross35]

@tgross35 I agree with what CAD97's said, I'm confused by your table and don't understand what you're trying to convey. Sized implies MetaSized (in fact I'm now convinced that it should be a supertrait of Sized) so all things that are Sized are MetaSized, but some things are not Sized but are MetaSized (eg trait objects), and finally extern types are neither.

I didn't put it together very well, but what I was going for was:

1. Anything with a Sized bound (doesn't specify ?Sized) will always be MetaSized - this is the default case
2. Something that is MetaSized may or may not be Sized, which is why you must specify ?Sized + MetaSized
3. Something that is strictly not Sized can never be MetaSized

Based on your comment it sounds like item 1 is implied (by MetaSized being a supertrait of Sized), item 2 is true, and I'm now realizing that item 3 is just contradictory to item 2 🙃.

What is the reasoning that optional traits aren't allowed in supertraits? Because ?Sized isn't adding a bound, it's opting out of one. When you write fn<T>, there's an invisible implied bound of T: Sized, unless you say that T: ?Sized.

Traits don't have this default implied bound of Sized, so ?Sized doesn't mean anything as a supertrait bound. Writing fn<T: JustSizedEnough> is the same as writing fn<T: Sized + JustSizedEnough>, and writing fn<T: ?Sized + Sized> is the same as writing fn<T: Sized>.

Thanks for the explanation. I know the discussion of trait aliases came up before, it just seems unfortunate that two traits ?Sized + MetaSized bound is now needed in many places where ?Sized works. But, I suppose that is what this step is meant to determine:

• [ ] Investigate how many ?Sized bounds would now need + MetaSized

[Skepfyr]

(This is mostly a brain dump so I don't forget it)

boats' post Changing the rules of Rust has made me realise that my plan of relaxing the meaning of ?Sized has backwards compatibility hazards that may prove problematic. Specifically any traits containing associated types can't obviously have the bounds on those associated types be weakened. This would be a massive problem for, e.g., the Deref trait, however I think we might be able to work around it by treating all pre-2024 code as implicitly having bounds like Deref::Target: MetaSized everywhere that they place a bound using a trait with an associated type.

Worse however is traits that contain functions that are generic over their argument because implementations of those traits could rely on T: ?Sized being MetaSized. However, with this doozy of a command:

jq '. as $root | .index[] | select(.crate_id == 0) | select(.inner | has("trait")) | (.inner.trait.items[] |= $root.index[.]) | select(.inner.trait.items[].inner.function.generics | .. | .modifier? == "maybe") | .name'

run over the standard library rustdoc json output, I believe the only stdlib trait this affects is RangeBounds because of its contains method. This is probably fine, but may present an issue still because it could force breaking changes to a bunch of ecosystem traits, for example serde's Serializer trait`, if they wanted to support these types.

[CAD97]

I got curious and ended up writing a partial implementation of ?MetaSized[^impl] for the current edition (i.e. ?Sized still implies MetaSized, but ignoring anything beyond the trait itself). My goal was to change as few library unbounds to ?MetaSized as are required[^unbounds]. I think my major takeaways got brought up thanks to boats' post, but as a short list:

[^impl]: I build the +stage1 compiler with (incomplete) ?MetaSized support, but then error during x check --stage 1 trying to derive Copy for ParamEnv, as for some reason it's determined that the std::marker::Copy impl for CopyTaggedPtr<&'tcx list::List<Clause<'tcx>>, ParamTag, true> requires that list::OpaqueListContents: std::marker::MetaSized (an extern type). I think it's actually the requirement P = &List<_>, P: Pointer, Pointer: Deref, since I still have Deref::Target: ?Sized; the odd error timing (the bound is structural on CopyTaggedPtr) is likely due to fast-accept paths for &_: Deref.

[^unbounds]: I only relaxed ?Sized to ?MetaSized for *const T, *mut T, &T, &mut T, NonNull<T>, PtrRepr<T>, PhantomData<T>, Unsize<U> (both U and Self), Pointee (Self), and the feature(ptr_metadata) fn.

• Current edition code should be able to write ?MetaSized. But I do generally agree that next edition code should have ?Sized unbound MetaSized as well and write ?Sized + MetaSized for current edition ?Sized.
• A decent amount of code assumes only one ?Trait unbound may be present.
• traits need a default Self: MetaSized bound/supertrait.[^supertrait] We elaborate supertrait bounds, so relaxing this implied bound on a trait should be semver-incompatible.{^supertrait-break]
• MetaSized is an object safe supertrait, so now most object safe traits fire warn(multiple_supertrait_upcastable) since the MetaSized supertrait is considered upcastable.[^upcast]
• When you write a generic list for<T: Deref>, this needs to be elaborated to essentially for<T: Deref<Target=$0>, $0: ?Sized + MetaSized>, where the ?Sized unbound is determined by ?Sized's presence on the associated item definition, but the MetaSized (un)bound is determined by ?MetaSized's presence in this bounds context, and this elaboration needs to be recursive.[^assoc]

[^supertrait]: I sort of do this -- adding a predicate alongside the Self: Trait predicate -- but with a huge hack where Unsize and Pointee are specifically opted out instead of looking for a ?MetaSized bound.

[^supertrait]: ...except that my implementation doesn't actually make the MetaSized bound visible in such a way that it gets elaborated currently, so it's accidentally semver-compatible to remove the non-elaborated supertrait bound.

[^upcast]: I just hacked around this problem by ignoring MetaSized in the lint. The better solution is to exclude MetaSized from dyn upcast eligibility... which actually the implied bound currently isn't permitted as a trait upcast by my impl, though it might still be put into the vtable; I didn't check.

[^assoc]: I was unable to figure out a way to do this bound elaboration yet, so my incomplete impl currently just doesn't. This is necessary because Deref::Target needs to be relaxed to ?MetaSized for basically anything to work.

[RalfJung]

When you write a generic list for<T: Deref>, this needs to be elaborated to essentially for<T: Deref<Target=$0>, $0: ?Sized + MetaSized>, where the ?Sized unbound is determined by ?Sized's presence on the associated item definition, but the MetaSized (un)bound is determined by ?MetaSized's presence in this bounds context, and this elaboration needs to be recursive.5

I cannot parse your syntax here and have no idea what you are talking about (not even parts of it), could you explain in more detail?

[CAD97]

It's easiest to explain by example:

// Assuming some trait associated type, e.g.
trait Deref {
    type Target: ?MetaSized;
    // ...
}

// when declaring the function
fn f<P: Deref>()
// this needs to be treated as having
where
    <P as Deref>::Target: MetaSized,
{
    // because the code might have previously relied on that bound, e.g.
    _ = size_of_val::<P::Target>;
}

// This application of implicit bound needs to be applied recursively;
// consider e.g.
trait Strategy {
    type Pointer: Deref
    where
        <Self::Pointer as Deref>::Target: ?MetaSized,
        // NB: bound would be implied here without unbound
    ;
}

// Then with a function defined as
fn g<S: Strategy>() {
    // we need both S::Pointer: MetaSized
    _ = size_of_val::<S::Pointer>;
    // and S::Pointer::Target: MetaSized
    _ = size_of_val::<<S::Pointer as Deref>::Target>;
}

It is technically an option to say that relaxing an associated type to unbound ?MetaSized is a major breaking change, but this is essentially untenable because Deref is critical to basic functionality which should continue to work for extern types. If we need to apply this implicit projected bound at preprojection bound declaration for existing traits anyway, it makes more sense to apply it to all trait projections than only some.[^mid]

[^mid]: There's an awkward middle ground where existing traits can opt into getting associated type MetaSized obligations elaborated, but new traits can have their associated types not get it. This enables traits to use impl detail associated types which are ?MetaSized without downstream current edition needing to unbound them. I don't even know if it's possible to make a type projection usable in your crate but not downstream without resorting to unreachable pub tricks, which would still get the implicit bound applied.

This actually makes me realize one more edge case to consider: trait DerefExt: Deref needs to have an implied bound of <Self as Deref>::Target: MetaSized as well, as default method bodies are currently able to (in effect) rely on the presence of that bound.

When to not apply the implicit bound is a bit more involved. If a generic binder context (params and where clause) introduces a trait obligation T: Deref (even for nongeneric T!), then an implicit obligation on the projection of <T as Deref>::Target: MetaSized needs to be introduced to the binder unless that same binder also contains a <T as Deref>::Target: ?MetaSized unbound[^unb] or we already have an equality constraint on the projection (typically written in source as T: Deref<Target = U>) in which case the implicit bounds on U of course apply instead. The recursive property is that <T as Deref>::Target may itself contain some positive obligations (Deref doesn't, but Strategy above does), any trait bound obligations introduced for the projected type also need this implicit bound to be added in order to ensure current edition code is never exposed to ?MetaSized generics/projections without writing the ?MetaSized unbound themselves.

[^unb]: Note that unbounds in this position (on a type other than a generic parameter introduced by that binder) are currently always a semantic error.

The extent of needing to apply MetaSized implicit bounds everywhere does make me wonder if it could be simpler to "just" "give up" and have the current edition code entirely unable to use ?MetaSized unbounds, with every reachable placeholder/generic type obligated to be MetaSized. It would be a strong cutoff between editions (and maybe even justify a distinctly settable dialect axis), but generically consuming ?MetaSized types can be expected to be rare. But I don't think applying a universal implicit obligation is really made significantly easier by not doing so in a way capable to being unbound. But I could see not allowing projection unbounds in the current edition, at least. Perhaps even supertrait unbounds[^sup]?

[^sup]: Supertrait MetaSized is actually one case where I'm wondering if it could be desirable to keep it as an implicit bound with an unbound even in the next edition. The reason is that the implicit bound is enabled to behave differently from an explicit bound; namely, be exempt from dyn upcasting (but allow it for an explicit bound for consistency) and not be elaborated (such that adding the unbound is nonbreaking), which is potentially desirable. On the other hand, traits do fine without an implicit Sized and adding Sized when necessary, so there's reasonably little reason to have a conservative MetaSized bound either.

Pessimistic takeaway: extern type support in the current edition is going to feel awkward no matter what we do, in order to forbid ?MetaSized types from "leaking". The best we can do is to make it acceptably predictable and at least somewhat usable, then clean it up to our regular standards for the next edition.

Making ?Sized the only unbound again does seem to be the better solution for the future, but with one potential exception I just thought of and want to point out: dyn Trait. Unowned &dyn Trait can be ?MetaSized just fine, but it would seem unfortunate to require that most owned trait objects specify Box<dyn Trait + MetaSized> instead.

[Jules-Bertholet]

This actually makes me realize one more edge case to consider: trait DerefExt: Deref needs to have an implied bound of <Self as Deref>::Target: MetaSized as well, as default method bodies are currently able to (in effect) rely on the presence of that bound.

You could gate the default method bodies behind the implied bound without gating the trait itself, as MetaSized would presumably be specialization-safe (no lifetime-dependent impls).

[snmps2]

Cool

[GoldsteinE]

There could be use in defining extern types with known alignment, like so:

extern {
    #[repr(align = 8)]
    type Foo;
}

This type can be the last field of a struct, but it’s not MetaSized, merely MetaAligned. Supporting such a feature would require separation of these two concepts, so I think they should be separated from the start to avoid having to migrate custom DSTs two times.

[WaffleLapkin]

@GoldsteinE unless I misunderstand what MetaSized/MetaAligned are, isn't Foo not just MetaAligned, but Aligned (i.e. the alignment is statically known even without metadata, similarly to slices).

Would also be nice to be able to specify "the same alignment as T", but that's besides the point.

[GoldsteinE]

@WaffleLapkin Sure, it’s also Aligned. MetaAligned still seems to me like a more natural restriction for a tail field, although I’m not sure MetaAligned + !Aligned types are even achievable without fully custom DSTs (i.e. DSTs with user-defined size_of_val and align_of_val).

Would also be nice to be able to specify "the same alignment as T", but that's besides the point.

I think that with generic_const_exprs it would be just Aligned<ALIGN = align_of::<T>()>.

[Skepfyr]

@nikomatsakis has written https://smallcultfollowing.com/babysteps/blog/2024/04/23/dynsized-unsized/ which suggests an alternative to relaxing ?Sized by using positive bounds (e.g. : MetaSized) that inherently relax the implied Sized bound. I'm sure I saw this suggestion before too but also have no idea where I saw it.

This suggestions should definitely go into the alternatives section at the very least, the way I see it the pros are:

• It doesn't require an edition boundary to make the change.
• It gets rid of the somewhat confusing ?Sized notation

The cons are:

• The implied relaxation of Sized could be surprising (especially if it works in supertraits), T: Unsized isn't really a bound at all, as it doesn't let the function assume anything about T making these traits especially weird.
• trait definitions currently assume Self is ?Sized + MetaSized (kinda? they allow functions that require Sized too which is odd), if this proposal means that in order to use a trait with an extern type it would have to be declared trait Foo: Unsized, it would be annoying to use them as it's likely crates would forget to do that.

[nikomatsakis]

trait definitions currently assume Self is ?Sized + MetaSized (kinda? they allow functions that require Sized too which is odd), if this proposal means that in order to use a trait with an extern type it would have to be declared trait Foo: Unsized, it would be annoying to use them as it's likely crates would forget to do that.

Hmm, I overlooked that, I think we would want to use an edition for this, or possibly we could just get away with changing the default to Unsized.