Amenities and ergonomics

[tokahuke]

So... I have been working with Cap'n Proto in Rust for some time by now and although I quite happy with it, I find myself writing a lot of boilerplate. How viable would it be for we to create abstractions based on Rust's macro system (thinking Serde, Rokcet, etc...) to make coding experience more fun?

Is this even the scope of any of the current crates?

I could help a bit on that.

[realcr]

I am working with capnproto-rust for a pretty large project, and I also found using capnproto-rust directly as it is somewhat difficult. I find that I always have to define the data at least three times:

1. capnp file
2. Rust structs
3. Code that converts capnp generated Rust structures to my Rust structs.

I realize that working like this introduces performance penalty (One of the major selling points of capnp is having no encoding/decoding step). However, If I want to enjoy the no-encoding feature I will have to surrender to capnp and let it pollute my whole codebase with lifetimes and .reborrow()-s. Sprinkle some async Futures into the mix and you get a real party.

I am willing to give up on performance in favour of clear and easy to maintain code. (At the same time I realize that other people might not want to).

To be more specific about what my problem is, I want to introduce an example from my code base:

1. capnp structure:

struct FriendsRoute {
        publicKeys @0: List(PublicKey);
        # A list of public keys
}

2. Rust structure

pub struct FriendsRoute {
    pub public_keys: Vec<PublicKey>,
}

3. Middle layer code, converting between capnp Rust structs and my Rust structs:

pub fn ser_friends_route(
    friends_route: &FriendsRoute,
    friends_route_builder: &mut funder_capnp::friends_route::Builder,
) {
    let public_keys_len = usize_to_u32(friends_route.public_keys.len()).unwrap();
    let mut public_keys_builder = friends_route_builder
        .reborrow()
        .init_public_keys(public_keys_len);

    for (index, public_key) in friends_route.public_keys.iter().enumerate() {
        let mut public_key_builder = public_keys_builder
            .reborrow()
            .get(usize_to_u32(index).unwrap());
        write_public_key(public_key, &mut public_key_builder);
    }
}

My plan is to have something like this:

#[capnp_conv(funder_capnp::friends_route)]
pub struct FriendsRoute {
    pub public_keys: Vec<PublicKey>,
}

Which will automatically do all the magic of producing the middle serialization layer (2), and also verify during compile time that the Rust structure has the same fields as the capnp structs. I plan to write it as a procedural macro.

Some inspiration for this plan is taken from the code I have seen in Facebook's Calibra, here: https://github.com/libra/libra/tree/master/common/proto_conv It contains a solution for a similar issue for protobuf ergonomics using a procedural macro.

I will start by writing the code as separate procedural macro crate for the Offst project. If things work out well, maybe we can add this idea into this repository's automatic code generation.

What do you think?

[dwrensha]

@realcr sounds good to me!

[tokahuke]

I have been using some of my own stuff under the hood, which boil down to basically the same thing. My setup is the following:

/// Makes the type an internal Rust representation of a Capnp.
pub trait AsCapnp {
    type CapnpRepr: for<'a> Owned<'a>;
    fn write_into<'a>(&self, builder: <Self::CapnpRepr as Owned<'a>>::Builder);
    fn read_from<'a>(reader: <Self::CapnpRepr as Owned<'a>>::Reader) -> Result<Self, capnp::Error>
    where
        Self: Sized;
}

Maybe I should better split it in two traits, like Serde does, but it has been work well so far.

I indeed have tried at first to force myself to use Cap'n Proto as natively as I could, but it always ended up being better to create an internal struct anyway. Besides, I have been working with the packed representation for most purposes (I am really in for the RPC part of the deal).

Oh! I have also these two macros for stub module generation:

#[macro_export]
macro_rules! stubs {
    ( $( $( $declaration:ident )* ; )* ) => {
        $(
            _stub! { $( $declaration )* }
        )*
    };
}

#[macro_export]
macro_rules! _stub {
    (mod $module:ident) => {
        pub mod $module {
            #![allow(unused)]
            include!(concat!(env!("OUT_DIR"), "/", stringify!($module) , ".rs"));
        }
    };
    (pub mod $module:ident) => {
        pub mod $module {
            #![allow(unused)]
            include!(concat!(env!("OUT_DIR"), "/", stringify!($module) , ".rs"));
        }
    };
}

stubs! {
    pub mod foo_capnp;
    mod bar_capnp;
}

These could also be of some use.

[realcr]

Hi, I made some progress with the suggested plan. At this point it should be possible to automatically serialize and deserialize Rust structures using capnp.

• Full code (Two separate crates)
• Test cases (To check how the syntax looks like)

Example

Here is a small snippet from one of the tests, to show the syntax. Capnp file:

struct FloatStruct {
    myFloat32 @0: Float32;
    myFloat64 @1: Float64;
}

Rust code:

use std::convert::TryFrom;

use offst_capnp_conv::{
    capnp_conv, CapnpConvError, CapnpResult, FromCapnpBytes, ReadCapnp, ToCapnpBytes, WriteCapnp,
};


#[capnp_conv(test_capnp::float_struct)]
#[derive(Debug, Clone)]
struct FloatStruct {
    my_float32: f32,
    my_float64: f64,
}

/// We test floats separately, because in Rust floats to not implement PartialEq
#[test]
fn capnp_serialize_floats() {
    let float_struct = FloatStruct {
        my_float32: -0.5f32,
        my_float64: 0.5f64,
    };

    let data = float_struct.to_capnp_bytes().unwrap();
    let float_struct2 = FloatStruct::from_capnp_bytes(&data).unwrap();
}

Currently supported

• All primitives except Void.
• Lists are supported, however, doing something like List(List(...)) is not yet supported.
• Structs
• Enums (Translated to capnp's Unions)
  • Unit variant is translated into Void
• Arbitrary nesting of Structs and Enums.

Things left to do

• Removing the requirement of having to import TryFrom, CapnpResult, ReadCapnp, WriteCapnp. Possibly by importing those anonymously inside the macro generated code. I'm still not sure how to do this.
• Handling error cases and compilation errors gracefully.
• Adding more comments
• Solving the nested List(List(...)) case.

I hope to complete some of those things in the following week. I also plan on trying to use capnp-conv inside Offst, to see that it works as intended.

Currently the code is part of the Offst project. I put the code inside the Offst repository because I can't afford to wait to have it available.

@dwrensha : capnp-conv has a license which is fully compatible with the license of capnproto-rust (MIT || APACHE at your choice). If you decide to add the code into the capnproto-rust project it can be added as separate two crates, without affecting the current code-base of capnproto-rust.

[insanitybit]

Any future progress on this? I feel that the macro approach is the best path forward. The existing approach from @realcr looks extremely promising, and imo should be an officially supported way to use capnp.

[realcr]

The full code for the macros approach can be found here (Coded as two crates): https://github.com/freedomlayer/offst/tree/master/components/capnp_conv

You have full permission to add it into this repository. (It has exactly the same license as this repository, so no paperwork is required). I will be more than happy to have capnp_conv as a dependency crate (:

I think that it is not perfect, and the code probably needs some polishing, but it works great for me. Hopefully it will be useful for other people.

[realcr]

@dwrensha: I added PR https://github.com/capnproto/capnproto-rust/pull/157, attempting to solve this issue.

I hope that the code quality of the additional crate is reasonable enough to be added to this repository. Please tell me if any modifications are required.

[jpochyla]

The code generated by capnproto-rust is un-idiomatic, hard to use (no support for sequences, frequent u32 casts, the type-as-module that confuses IDEs, ...) and slow (bounds checks in sequences, the scratch space foot gun). Why isn't this talked about more? In most cases, the capnp types won't match the internal, domain, types anyway, so an automated macro-based lipstick won't really help.

Edit: It's much better with the improved setter ergonomics from the 0.19 version. Thank you @dwrensha!