Allow UniFFI exported types (e.g. `Object`) from EXTERNAL crates to be used in a crate's e.g. `Record`

[Sajjon]

@bendk asked my in this Matrix thread to open a GH issue for this

Context

I'm writing a lib which must fully control Serde and UniFFI exporting of all types, this involves a lot of newtype pattern and usages of uniffi::custom_newtype!, so that I can implement custom Serde for it.

However, one type is quite complex and I would like to write as thin a wrapper as possible around an external type, a (Big)Decimal type. Radix Engine Toolkit's (short: RET) UniFFI crate vendors a Decimal type, lets call it RETDecimal, which is UniFFI exported as Object, it itself actually just wraps the Decimal type from another crate (another repo): Radix Scrypto Decimal lets call it NativeDecimal.

NativeDecimal in Scrypto repo knows nothing about UniFFI.

My lib lives in another repo than RETDecimal and another repo than NativeDecimal (perhaps only crate hierarchy is relevant, not repo...).

What I would like to have is something like:

#[derive(uniffi::Object)]
pub struct Decimal(AnyOfTheOtherTwoDecimalTypes);

#[uniffi::export]
impl Decimal {
    pub fn add(&self, other: Self) -> Self { ... }
    pub const fn zero() -> Self { ... }
   // ... + 100 other methods and functions
}

Where by AnyOfTheOtherTwoDecimalTypes I mean I'm happy with using already UniFFI exported RETDecimal from RET crate, or the wrapped type it uses, NativeDecimal from the Scrypto crate (which does not know anything about UniFFI).

or alternatively `Record:

#[derive(uniffi::Record)]
pub struct Decimal {
    inner: AnyOfTheOtherTwoDecimalTypes,
}

And the same API - as in exposed functions and methods.

It is important that the Decimal type must not be a typealias to a primitive in the Swift (or Kotlin) land, it must be its own type. I do not care about the fields of it, so it can be an uniffi::Object - however in the Rust land using Object requires usage of Arc, which breaks derive of Hash and Eq of an owning holder of the type... why I prefer uniffi::Record.

Yes I know, Records cannot have methods, it is OK, I would create global freestanding functions and in the Swift land I would create extensions:

extension Decimal { 
    func add(other: Self) -> Self { 
        // calling UniFFI exported global freestanding method since `Decimal` is a `Record` which does not support methods
        addDecimal(self, other)
    }
}

Failed attempts

As seen in the Matrix thread, I've tried uniffi::custom_newtype!(Decimal, RETDecimal); which did not work. I also tried:

impl crate::UniffiCustomTypeConverter for Decimal {
    type Builtin = RETDecimal;
    ...
}

Did not work.

And I've tried:

unsafe impl<UT> Lower<UT> for Decimal {
 ...
}

Did not work.

Then as per suggestions I've tried:

[ExternalInterface="radix_engine_toolkit_uniffi"]
typedef extern Decimal;

But did not get that to work either.

Hopefully there is a combination of any of the three + [Extern] in UDL which will work that I have not yet found.

[bendk]

I think these approaches should work:

• The ExternalInterface in a UDL file you laid out
• Importing the type (use RETDecimal from radix_engine_toolkit_uniffi)
• Defining the Decimal(NativeDecimal) struct with methods that you can use (Isn't this the exact way you created the RETDecimal type?)

What error message to you get when you try those.

The best way to get help for things like this is to create a minimal example in a uniffi branch and push it out so we can investigate. This gets tricky with multiple crates involved, so alternatively you could create branches of your repos that show the issue and I can take a look.

[Sajjon]

Thx! I will give it a go

[CyonAlexRDX]

(This is @Sajjon under work Github username) @bendk is it possible to "bulk re-export" a crate? What do I mean by that? The radix_engine_toolkit_uniffi (short: RET) crate is quite big, the resulting header file is 2000 lines. Above it sounds like I have to define each RET type in the UDL of my crate Sargon (repo: RadixWalletKit)? Also hmm how does it work with functions inside of RET? Is it possible to re-export RET's uniffi exported functions with my crate Sargon?

Instead of having to put an entry for each RET type (and function?) in Sargon's UDL, is it somehow possible to in Sargon say "I wanna RE-EXPORT the whole of RET", so that all consumers of Sargon's UniFFI also gets RETs symbols (types and functions)?

Maybe my question does not make sense, sorry, I have a bad understanding of how a crate with UniFFI support can re-export/re-vendor another crate also with UniFFI support :).

[CyonAlexRDX]

I might in fact want to not directly re-export RET's types and functions... but rather translate its API to Sargons own API with Sargon owned types... but I'm still curious as to if it is possible.

[bendk]

Instead of having to put an entry for each RET type (and function?) in Sargon's UDL, is it somehow possible to in Sargon say "I wanna RE-EXPORT the whole of RET", so that all consumers of Sargon's UniFFI also gets RETs symbols (types and functions)?

Re-exporting everything is currently not possible, but the way we generally handle that issue is to run the bindgen against both crates, generating 2 foreign modules. With Python, the resulting code looks something like this:

from radix_engine_toolkit import RETDecimal
from swift_engine_toolkit import Decimal

This works with Swift as well, but there's a couple gotchas:

• All names go in the global namespace, so you need to make sure their unique across both crates. We've discussed adding an option in uniffi.toml to prefix names and/or rename particular types in Swift, but haven't implement this.
• The header files and/or module maps need a bit of care to combine together correctly, I forget exactly what the trick is but I don't think it's that hard. I've been hoping to improve the DX for this when you run uniffi in library mode, but again haven't gotten around to implementing it.

[bendk]

With the approach above, you still need to flag types used across crates in the UDL file with [External], but that's usually much fewer types.

[mhammond]

There's some interest around external/remote/custom types recently, so it would be great to see this looked at again. I'm struggling to work out what's being asked for exactly here though - maybe a minimal code-only sketch of the problem? I'm going to close this now but would love to find out what the ask here is for exactly in another issue.