Serialization of stream graph

[Widdershin]

It would be very useful if we could serialize and deserialize both xstream observables and observable graphs.

This would enable snapshotting of application state, which opens up a world of possibilities around crash log reporting, persisting application state, enabling cycle-restart to replay from a checkpoint.

From a code standpoint, this would look something like this:

const stream = xs.periodic(5000);

const listener = stream.addListener({
  next: i => console.log(i),
  error: err => console.error(err),
  complete: () => console.log('completed'),
});

let serializedStream;

stream.skip(4).take(1).addListener({
  next: i => {
    serializedStream = stream.serialize()
    stream.removeListener(listener)
  },
  error: err => console.error(err),
  complete: () => console.log('completed'),
})

console.log(JSON.stringify(serializedStream)) // should work

const deserializedStream = xs.deserialize(serializedStream);

deserializedStream..addListener({
  next: i => {
    assert.equal(i, 5)
  },
  error: err => console.error(err),
  complete: () => console.log('completed'),
})

[Widdershin]

As discussed, in order to avoid circular data structures in the serialized format, we would probably need to normalize the data into something like this:

{
  streams: {
    a: {
      kind: 'periodic',
      ...
    },
    b: {
      ...
    }
  },

  dependencies: [
    {from: 'a', to: 'b'}
  ]
}

[cem2ran]

Looks like the Rx people are already working on this but for a far more complex use case. As explained in the following talk they are utilizing an intermediate expression tree format called bonsai to ship a stream graph to a remote context, e.g. a server: https://vimeo.com/132192255 and a little more chatter about it here: https://gitter.im/Reactive-Extensions/Rx.NET/archives/2015/07/01

RxJS project seems to be implementing this as well but doesn't seem to be completed yet: https://github.com/Reactive-Extensions/RxJS/blob/master/src/core/expressions

Should lambda expressions also should be serialized, e.g. .map(fn)?

[staltz]

@cem2ran https://github.com/ReactiveX/rxjs/issues/1272 ;)

[Widdershin]

I wonder how you would go about doing this without serializing selector functions. The issue there is that if you've closed over anything then you can't really get away with serializing the function. Hmmmmm

[staltz]

@Widdershin I think we don't need to serialize everything, it would be a necessity only when doing things like transferring a stream graph over the network to another computer.

Our use case for serialization is primarily for snapshotting. So it's enough to snapshot the state that each operator contains, so when the application is restarted, it will still have the same stream graph, we just need to load all the internal state onto it. Or am I missing some corner case?