no_std support

[cbeck88]

Thanks for this library!

Is there any interest from the devs in no_std support?

We have been using corepack as an alternative but it is inferior to rmp in a number of ways unrelated to the std support.

I might be willing to provide a patch along the lines of,

1. adding a feature nightly which will allow using nightly-only features like alloc crate
2. selecting serde and byteorder with default-features = false in the Cargo.toml in nightly mode
3. Replacing most uses of std prefix with either core or alloc as appropriate

[3Hren]

Hi! I have no experience in no_std adaptation. However, it may be useful for someone else. Feel free to open a PR!

[wagnerma]

Hi, any updates on this, I am also intersted in support for no_std. My rust know-how is still to low to help coding, but testing would be possible.

[kornelski]

I'm happy to take pull requests for this.

[cbeck88]

wow, it's been a long time...

So unfortunately, my conclusion after studying over the past 1+ year is that rust serialization libraries based on std::io are unsuitable for porting to no_std environments. std::io is very very difficult to port in any maintainable way. The most serious attempt seems to be this. https://github.com/jethrogb/rust-core_io

The serialization libraries that I recommend for no_std + alloc environments are:

• serde_cbor which is a great crate! https://github.com/pyfisch/cbor
• prost, which is based on tokio::bytes instead of std::io, and so is relatively easy to port to a no_std + alloc environment. I have a LONG running PR here: https://github.com/danburkert/prost/pull/215 which I am extremely hopeful will finally be merged

[daboross]

Thanks for investigating this!

I haven't investigated this fully, but might it be reasonable to port to no_std if we added a layer of abstraction between our read/write requirements & std::io? If we could break our dependency on std::io::Read and std::io::Write in no_std environments, then I think we could support it.

I'm imagining something like

pub trait Read {
    fn read(&mut self, buf: &mut [u8]) -> Result<usize>;
    fn read_exact(&mut self, buf: &mut [u8]) -> Result<()>;
}
#[cfg(feature = "std")]
impl<T> Read for T where T: std::io::Read { ... }
#[cfg(not(feature = "std"))]
impl<'a> Read for &'a [u8] { ... }

It'd be more abstraction - but if we did that, and something similar for write, could we bypass the ickyness of std::io?

If that is reasonable, then I think supporting no_std should be fully possible. alloc is stable now, so if we bump our MSRV for non-default-features to 1.36.0, then we could add this without a nightly feature and just have an optional-but-default std feature. I don't have the time atm to implement this, but I think it should be possible?

[cbeck88]

@daboross I picked up someone else's fork of your library to try to create the shims that you are talking about, almost 18 months ago.

Here's what I had: https://github.com/garbageslam/msgpack-rust/tree/no_std_support

We worked with that internally for about 12 months. Later I tried to rebase on current msgpack-rust and there were enormous amounts of conflicts, some of it having to do with unsafe code, and also like, changes in rustlang itself. That was when we dropped this and migrated to serde_cbor.

Here's the guy's branch I started from, which was developed 3 years ago: https://github.com/pftbest/msgpack-rust/tree/no_std

So I think it's safe to say that people have been trying to create nice portable abstraction layers over std::io for many years now, without much success. If you succeed, it will be a great service to the community!

---

I think the biggest problem is that std::error::Error is not portable. Even though it has no dependencies on mutexes, or threading, or anything like that, it cannot be moved to core, or even alloc crate, due to tech debt in the standard library. And due to the way rust works, if the traits themselves are non-portable, not merely the implementations, you can't really work around that at all. The fundamental traits have to be in core to be useful at all, if the goal is to write portable code.

Since std::io uses this everywhere, that's a big stumbling block right out of the gate.

The tokio::bytes crate is different -- bytes represent a buffer that you already have, and can touch in lots of ways without it posisbly being a system call. There are lots of "no-fail" operations there and its easy to use cleanly without errors flying out everywhere. I think that being able to deftly avoid std::io is one of the biggest advantages that prost has.

I haven't studied the serde_cbor internals enough to understand how they deal with this, because serde also depends on std::error::Error, when std feature is on... serde itself has been a big portability headache honestly. I complained about this once and dtolnay made this release: https://github.com/serde-rs/serde/releases/tag/v1.0.100

[Techcable]

This would be (partially) fixed by #295 😄

Only thing to note is RmpWrite/RmpRead are considered an implementation detail in that PR.

That means on #[no_std], the only supported types would be Vec<u8> for writing and &[u8] for reading.