Constant memory API server history using StateEvents

[ch1bo]

Why

While working on the hydra-doom project, we noticed that the in memory state grew without bound (see #1572). This meant that, at the sustained load that the hydra doom demo was producing, nodes became inoperable after just a few hours.

One of the culprits was the ?history of the hydra-node websocket API. That feature sends clients the whole history by default (can be turned off, see documentation https://hydra.family/head-protocol/docs/api-behavior#replay-of-past-server-outputs)

We want to make Hydra heads be operable for long times even with sustained load, which requires us to bound any memory usage (and even disk usage, see #1581).

What

• The default behavior of the websocket API is to not send the whole history, but only the Greetings message.
  • This breaking change is to be communicated properly
• The API server history, if requested by ?history=true, is loaded on-demand from disk.

How

• [ ] #1717
• Remove ServerOutput type and add/create constructors in StateChanged
  • Most ServerOutputs directly map to a StateChanged event
  • Some additional events need to be created to server outputs like PeerConnected et al
• Instead of cause ClientEffect, the HeadLogic would emit events with newState. Example: https://github.com/cardano-scaling/hydra/blob/f7703190f83f3af51350189cfdc237bbff858f5e/hydra-node/src/Hydra/HeadLogic.hs#L599-L600 would become

  newState SnapshotConfirmed{headId, snapshot, signatures = multisig}
  

• Refactor the API Server
  • API-internal Projection read models need to aggregate over StateEvent
  • Re-use eventId as sequence number
  • Load events on startup and on-demand to initialize read models and next sequence number.
  • Load events on-demand if a new connection wants the full ?history
• Now the API server is just an EventSink
• Websocket needs to emit this information on new state changes