`Arc<impl !Sized>` constructors

[Nemo157]

Proposal

Problem statement

Given some unsized type T it's not possible to create an Arc<T> except for some special cases supported in std (e.g. Arc::new_uninit_slice).

Motivating examples or use cases

In a private library we want to allocate a shared data structure with some header and a dynamic length slice of additional data:

#[derive(Debug)]
struct SharedData {
    header: Header,
    flags: [AtomicUsize],
}

let shared: Arc<SharedData> = ...;

Solution sketch

After a little iteration the best API I came up with looked like (full playground)

impl Arc<T: ?Sized> {
    /// Allocate the space for an `Arc<T>` where the `T` value has the given metadata.
    /// The returned pointer points to uninitialized memory and must be initialized before calling `Arc::<T>::from_raw` to convert into an arc.
    unsafe fn raw_new_with_metadata(metadata: <T as Pointee>::Metadata) -> *mut T;
}

From what I can tell there is no way to provide an API using only stable interfaces, Metadata is necessary for making the pointer.

At first I wanted to return Arc<MaybeUninit<T>>, but it's not possible to have a MaybeUninit<impl !Sized>, if that becomes possible in the future it would likely be a better type signature.

Alternatives

Currently afaik there are only two alternatives: split the data into two Arcs so the special-case constructors can be used, or add extra indirection for the tail so that the type becomes Sized. Both of those have performance downsides.

This cannot be (soundly) implemented outside std because the details of how to get a pointer valid for Arc::from_raw are internal.

Links and related work

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[Nemo157]

One issue with this API I've had pointed out is that it doesn't handle deallocation if initialization fails; and even if there were additional APIs to support that it is tricky with unwinding.

One suggestion is to have the initialization happen in a callback, so the Arc impl is responsible for having a drop-guard and deallocating the memory if it fails

impl Arc<T: ?Sized> {
    /// Creates a new `Arc<T>` where the `T` value has the given metadata.
    ///
    /// The given metadata must be valid for `Layout::for_value_raw`
    /// (except `extern type` being allowed to return the wrong thing complicates it).
    ///
    /// The `init` function must fully initialize the value behind the pointer.
    /// If initialization fails (either error or unwinding) the `init` function must drop
    /// any fields it has written so far (or they will be leaked), this function will
    /// take care of deallocating the backing memory.
    unsafe fn new_with_metadata<E>(
        metadata: <T as Pointee>::Metadata,
        init: impl FnOnce(*mut T) -> Result<(), E>,
    ) -> Result<Self, E>;
}

This wouldn't be an issue if we somehow get unsized MaybeUninit so it can simply return an Arc<MaybeUninit<T>> that gets assume_inited once written to.