Efficient (de)serialization of gluon values from Rust

[theduke]

Hi there.

I'm considering Gluon as the primary embedded language in a event sourcing server.

Users would define event types and projections (basically a fold over events) in Gluon.

These would be received by the server in some format, passed to Gluon code where validation and folding would happen. Then these types would have to be serialized to some format like bincode for persisting them in some underlying storage.

So, to clarify, an example flow:

1. server receives data in a serialization format (json, bincode, ...)
2. data is deserialized to types defined by gluon code
3. Gluon code works with the data and potentially produces new data
4. The resulting data is then serialized by Rust code and stored somewhere

Is there a way to do this reasonably efficiently?

Is there any kind of integration between Gluon and serde? Or are there any other suggestions on how to handle this?

[Marwes]

Depending on the specific needs here this might all work already or may take a bit of work to get working nicely.

If you know the shape of the data you take as input and output well enough to define rust structs for it then there shouldn't be any issue going serialization format -> rust struct -> gluon value -> rust struct -> serialization format (HashMaps don't work out of the box though #514 ).

If the shape is dynamic I have just added JSON support which works fairly well #586. Other self-describing could be added without much work as well but non-self describing formats like bincode won't work out of the box.

Is there a way to do this reasonably efficiently?

So it depends, there is an intermediate layer for JSON right now but it is something I'd like to eliminate (where it is possible). It could be rather complicated however so it could take some time to get done.

Is there any kind of integration between Gluon and serde?

There is an integration to serialize gluon values to and from self-describing formats using serde at https://github.com/gluon-lang/gluon/blob/master/vm/src/api/ser.rs and https://github.com/gluon-lang/gluon/blob/master/vm/src/api/de.rs . However this API is only usable from Rust (not pure gluon) as it needs to deal with gluon types.

[Snarpix]

Is it currently possible to create gluon Json value from rust? My goal is to pass serde_json::Value to gluon, without to_string and deserialize steps.