crABI v1

[joshtriplett]

Note that the most eminently bikesheddable portion of this proposal is the handling of niches, and the crABI Option and Result types built around that. There are multiple open questions specifically about that.

I've also listed an open question about how to represent owned crABI pointers as Rust types: Box<T> versus Box<T, NoDeallocate> versus Box<T, FFIDeallocate<obj_free>>.

Rendered

[Lokathor]

The "rendered" link seems to point at a file that's gone after the force push

[programmerjake]

for Box<T, A>, we could introduce a new trait:

// in std
pub trait BoxDrop<T: ?Sized>: Sized {
    fn box_drop(v: Pin<Box<T, Self>>);
}

impl<T: ?Sized, A: Allocator> BoxDrop<T> for A {
    #[inline]
    fn box_drop(v: Pin<Box<T, Self>>) {
        struct DropPtr<T: ?Sized, A: Allocator>(*mut T, A, Layout);
        impl<T: ?Sized, A: Allocator> Drop for DropPtr<T, A> {
            #[inline]
            fn drop(&mut self) {
                if self.2.size() != 0 {
                    unsafe { self.1.deallocate(NonNull::new_unchecked(self.0).cast(), self.2) }
                }
            }
        }
        let l = Layout::for_value::<T>(&v);
        let (p, a) = Box::into_raw_with_allocator(unsafe { Pin::into_inner_unchecked(v) });
        let v = DropPtr(p, a, l);
        unsafe { v.drop_in_place() }
    }
}

// the standard Box type
pub struct Box<T: ?Sized, A: BoxDrop<T> = Global>(...);

// replacement version of Drop for Box
impl<T: ?Sized, A: BoxDrop<T>> Drop for Box<T, A> {
    #[inline]
    fn drop(&mut self) {
        A::box_drop(unsafe { ptr::read(self) }.into_pin())
    }
}

usage demo:

pub struct FooDropper;

impl BoxDrop<Foo> for FooDropper {
    fn box_drop(v: Pin<Box<Foo, FooDropper>>) {
        drop_foo(v);
    }
}

extern "crabi" {
    pub type Foo;
    pub fn make_foo() -> Pin<Box<Foo, FooDropper>>;
    pub fn drop_foo(v: Pin<Box<Foo, FooDropper>>);
}

[joshtriplett]

@programmerjake That's an interesting alternative!

Is there a way, rather than having to implement a trait, to instead have a single type parameterized with a function type?

[programmerjake]

Is there a way, rather than having to implement a trait, to instead have a single type parameterized with a function type?

I thought about it, but it's very annoying to give function types a name, since you currently have to use TAIT:

struct FnDropper<T: ?Sized, F: Fn(Pin<Box<T, FnDropper<T, F>>>)>(F);

type FooDropFn = impl Fn(Pin<Box<Foo, FnDropper<Foo, FooDropFn>>>)
extern "crabi" {
    type Foo;
    fn make_foo() -> Pin<Box<Foo, FnDropper<Foo, FooDropFn>>>;
    fn drop_foo(v: Pin<Box<Foo, FnDropper<Foo, FooDropFn>>>);
}
#[defining(FooDropFn)]
fn _f() -> FooDropFn {
    drop_foo
}

[programmerjake]

maybe better usage demo:

// std API
pub struct FFIDropper;
// like C++ unique_ptr but where deleter is defined by T
pub type FFIBox<T: ?Sized> = Box<T, FFIDropper>;

// user API, this impl could easily just be a
// #[box_drop = drop_foo] proc-macro annotation on Foo
impl BoxDrop<Foo> for FFIDropper {
    fn box_drop(v: Pin<FFIBox<Foo>>) {
        drop_foo(v);
    }
}

extern "crabi" {
    pub type Foo;
    pub fn make_foo() -> Pin<FFIBox<Foo>>;
    pub fn drop_foo(v: Pin<FFIBox<Foo>>);
}

[programmerjake]

lots more discussion about BoxDrop and FFIBox and stuff here: https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/BoxDrop.20proposal/near/383840871

[joshtriplett]

@programmerjake I attempted to partially summarize that proposal in the alternatives section. I do agree that if we accepted that general proposal, it makes sense to use it for the specific case of crABI's handling of Box.

[EdorianDark]

Provide the initial version of a new ABI and in-memory representation supporting interoperability between high-level programming languages that have safe data types.

Are there other languages interested in this proposal? Or is the target to enable an ABI between Rust code?

[Araq]

There is interest from Nim (disclaimer: I'm Nim's BDFL). But for Nim it would be really nice if "bit flags" which Nim maps to its set construct could become part of the spec. (Rust can only do the terrible low level bitwise operations here.)

Also quite discouraging is the lack of Swift support, IMO. A common ABI for subsets of Nim, Rust, Swift and C++ seems quite feasible.

[joshtriplett]

Part of the goal here is to have a baseline level of support from any language that speaks C FFI, which then means that any language with a C FFI immediately has the ability to interoperate with crABI. Everything beyond that is then about the convenience of native support (whether language or library), rather than about whether it's supported or not. So, anything supported would need to map to an underlying C data type that can be passed through the C ABI.

(I do expect that eventually we'll want to support a full object/trait protocol, but I'm trying to get there incrementally rather than trying to do it all at once. The initial round of crABI support is optimizing for ease of initial support/adoption, rather than completeness.)

@Araq This mechanism: https://nim-lang.org/docs/manual.html#set-type-bit-fields ? (Verifying: does Nim support bitfields wider than a base integral data type, or do they have to fit in a base integral data type?) That seems like a reasonable data type to support cross-language. At a minimum, that seems like an ideal candidate for crABI v1.1 (which I expect to follow closely on the heels of crABI v1.0).

[Araq]

@Araq This mechanism: https://nim-lang.org/docs/manual.html#set-type-bit-fields ?

Correct.

(Verifying: does Nim support bitfields wider than a base integral data type, or do they have to fit in a base integral data type?

It supports bitsets wider than any integral data type indeed. But an ABI could limit it to an integral type.

[tmccombs]

Given that this is intended for cross-languange interfaces there should probably be a formal, mostly language agnostic specification of the ABI. Should this RFC discuss where that specification should go, and how it will be created? Will maintainers from other languages (such as nim) be involved in that process?

[comex]

I suggest adding to the RFC that crABI will not initially define a stable LLVM CFI mangling (see also #3296, ping @rcvalle).

For an example of where this would be an issue, it's one thing to say that (quoting the RFC):

extern "crabi" fn func(buf: &mut [u8]);

is equivalent to:

struct u8_slice {
    uint8_t *data;
    size_t len;
};
extern void func(struct u8_slice buf);

But under LLVM CFI, if func ends up being turned into a function pointer, it will get a hash based on the Itanium C++ name mangling of the function signature, which has to match between caller and callee ends. On the C side, it would be mangled based on the actual name of the struct (in this case, u8_slice). On the Rust side, slices are currently mangled as vendor extended types, which can't be expressed in C or C++.

One potential solution is to define a standard C++ equivalent name for CFI purposes, e.g. [u8] could be mangled as if it were a C++ type named ::rust::slice<u8>. That would help when binding to C++, but not when binding to C, let alone other languages.

Another potential solution is to define a standard C struct name (say, _rust_slice_u8), but that's just really gross (how would it deal with more complex types? manually writing out the mangling in the struct name?), and also wouldn't work well with C++.

I think a better approach is to add an attribute to Clang to customize the CFI mangling for a C struct definition. After all, I'm not aware of any other implementations of LLVM CFI besides rustc and Clang, so it's not critical for us to fit into the existing constraints. We could then either keep using vendor extended types or go with the C++ equivalent name; which approach works better would depend on the design of said attribute.

But I'm not volunteering to send a patch to Clang; I think that onus is on anyone who wants to use crABI in the context of CFI-protected (C/C++)-Rust interop.

If that doesn't happen soon, though, then what? Clearly there's no need to block anything about crABI not related to CFI. The question is whether it should block stabilizing the LLVM CFI mangling. There's an argument that it shouldn't: we could stabilize the CFI mangling as-is and it would still be useful to protect Rust-to-Rust calls. But given the desire to interoperate well with other languages, I think it would be better to wait on stabilizing the CFI mangling until we have a full answer for how it will interop with Clang.

[Araq]

Please leave name mangling unspecified until interop between different languages has been established in some prototypes. It also does not have to be part of crABI at all and could be a different spec altogether.

[nortti0]

One way to reduce the complexity of the niche optimizations would be to only use two values for the niche value:

• Option<&T>, Option<&mut T>, Option<NonNull<T>>, Option<Box<T>>, and Option of any function pointer type are all passed using a null pointer to represent None.
• Option of any of the NonZero* types is passed using a value of the underlying numeric type with 0 as None.

These can continue to use the zero value for None.

• Option<bool> is passed using a single u8, where 0 is Some(false), 1 is Some(true), and 2 is None.
• Option<char> is passed using a single u32, where 0 through 0xD7FF and 0xE000 through 0x10FFFF are possible char values, and 0x110000 is None.
• Option<OwnedFd> and Option<BorrowedFd> are passed using -1 to represent None

These could all use the all bits set (i.e. 0.wrapping_sub(1)) value for None, as is already used for the *Fd types.

A second, further way to simplify the specification here and allow users to enable the niche optimization for their own types would be to define a NonNegativeI* (other spellings are possible, e.g. u31) series of types. Types that are currently called out here explicitly would instead be defined to be passed using those types, and the niche optimizations section could be changed to read:

• Option<&T>, Option<&mut T>, Option<NonNull<T>>, Option<Box<T>>, and Option of any function pointer type are all passed using a null pointer to represent None.
• Option of any of the NonZero* types is passed using a value of the underlying numeric type with 0 as None.
• Option of any of the NonNegative* types is passed using a value of the underlying numeric type with all bits set as None.
• Option of a repr(transparent) type containing one of the above as its only non-zero-sized field will use the same representation.

[programmerjake]

A second, further way to simplify the specification here and allow users to enable the niche optimization for their own types would be to define a NonNegativeI* (other spellings are possible, e.g. u31) series of types. Types that are currently called out here explicitly would instead be defined to be passed using those types, and the niche optimizations section could be changed to read:

that wouldn't work for (some of) the Win32 handle types, because iirc negative handles are perfectly valid, it's just -1 that's invalid.

[ChrisDenton]

that wouldn't work for (some of) the Win32 handle types, because iirc negative handles are perfectly valid, it's just -1 that's invalid.

-1 is perfectly valid. It's a pseudo handle for the current process. But yes, negative handles are valid; they're static values with special meaning. The only common niche between real handles and pseudo handles is 0.

[programmerjake]

that wouldn't work for (some of) the Win32 handle types, because iirc negative handles are perfectly valid, it's just -1 that's invalid.

-1 is perfectly valid. It's a pseudo handle for the current process. But yes, negative handles are valid; they're static values with special meaning. The only common niche between real handles and pseudo handles is 0.

I'm referring to OwnedSocket which has a niche for INVALID_SOCKET aka. -1 because that is not a valid socket handle.

[matklad]

Hm, I am quite a bit surprised that the RFC doesn’t talk about aliasing at all. Consider this crabi function declaration:

struct u8_slice {
    uint8_t *data;
    size_t len;
};
extern void copy(struct u8_slice dst, struct u8_slice src);

is the code calling the function required to ensure that src and dst do not overlap? Is the code implementing the function allowed to assume that the slices do not overlap?

Do we just use Type Based Alias Analysis rules here by virtue of just deferring to C ABI?

[bjorn3]

is the code calling the function required to ensure that src and dst do not overlap?

If the rust side is &mut [u8] then yes. If it is *mut [u8] then no. In general I did expect the regular rust memory model rules to apply.

Do we just use Type Based Alias Analysis rules here by virtue of just deferring to C ABI?

TBAA is entirely incompatible with rust.

[matklad]

TBAA is entirely incompatible with rust.

Yes, but, if I understand this right, that's what the current RFC implicitly proposes to use, by saying that "we lower to C ABI".

Taking example from https://stefansf.de/post/type-based-alias-analysis/, the following crabi declaration:

extern void foo(int *x, short *y)

also carries implicit constraint that x and y do not alias (level of confidence: 0.7).

We definitely don't want to have that in crabi, because that's not how the Rust works, and not how an ideal ABI would work. But that means we need to explicitly define aliasing rules for crabi, as otherwise we are inheriting those from C.

[jonathanpallant]

If I have some crabi function which takes, say, core::crabi::Option<u32> as an argument, can I pass Some(42) and have it auto-convert (or auto-infer), or do I have to say core::crabi::Option::Some(42) (or Some(42).into())?

I found the manual conversions was one of the annoying parts about using the C ABI in https://github.com/Neotron-Compute/Neotron-FFI/blob/develop/src/option.rs (e.g. here or here)

But also, yay, I could basically delete the neotron-ffi crate.

[comex]

extern void foo(int *x, short *y)

also carries implicit constraint that x and y do not alias (level of confidence: 0.7).

You don't get UB in C just by having pointers that alias, only if you actually dereference them with incompatible types.

So the full example from the post you linked has UB if x == y:

void foo(int *x, short *y) {
    *x  = 40;
    *y  = 0;
    *x += 2;
}

But this version would not, as long as the pointee was originally ether a dynamic allocation or a variable of declared type int:

void foo(int *x, short *y) {
    *x  = 40;
    memset(y, 0, sizeof(short));
    *x += 2;
}

As a result, there's no need for crABI to, say, munge the types to void * at the boundary. As such, I think the issue is largely out of scope for crABI.

A potential exception is that if a pointer to a Rust local or global variable is sent to C, the C side might want to know what the "declared type" is for the purpose of C's aliasing model. The right answer should be that the Rust variable is like a dynamic allocation and doesn't have a declared type. But I'm not sure if that works in all cases in the current implementation, or if it should be guaranteed for all Rust implementations. Still, it seems somewhat out of scope…

[Maix0]

I would actually like to see a deny by default lint&compiler flag (which is a new thing I think) that would (when allowed/warned) make the transparent conversion between crabi::Option and std::Option (and the result type as well).

As this would be an deny by default, you wouldn't fall into relying on it by accident, but would still allow someone to quickly prototype/get/use an FFI API quickly.

I do feel strongly against having this in everyday program, as it is indeed basically a deep memory copy, but this could allow some leeway when trying stuff.

It is true that having to duplicate those types is sad, but as said in the RFC, there seems to be no other way.

[jeffparsons]

It is true that having to duplicate those types is sad, but as said in the RFC, there seems to be no other way.

I know I'm just one data point, but as a Rust user who is excited about crABI: I get it. I'll live. Better a bit of minor friction than lurking dragons.

And it seems to me that it's pretty easy to leave the door open to creative solutions in future, so at least to me it doesn't seem like solving it now should be considered a blocker.