Serialize all messages Middleware

[rogeralsing]

Let's add a middleware in Remote, that allows us to serialize all messages.

[DenizPiri]

We implemented something like this in our codebase. Some notes of our usage:

• We only use it for Cluster actors.
  • We have many locally spawned actors and we are not interested in making their messages serializable. We just want them to be immutable.
• Instead of serializing all messages, we serialize only a given percentage of them (based on a random dice roll).
  • This allows us to test both the cases of cluster actors being remote and being local.
  • If we were to serialize all messages, we could easily make a mistake by sending a mutable message and actually mutate it after sending; the error wouldn't show up in testing.
• Instead of serializing/deserializing during receive, we actually do it during sending (we use Cluster.RequestAsync via our wrapper which does this).
  • We do it this way because our Cluster actors usually send Timer messages to self, and those again are not meant to be serializable.
• In our RequestAsync implementation, we do this not just for the message sent, but also the response received, as both must be serializable.
• We run the serialization chance at 50% for local dev, unit tests, and staging deployments. Only at production deployments, it is fully disabled. This allowed us to catch many issues that would only show up in production immediately in unit tests, without having to convert all our tests to be multi-member.

[seungyongshim]

Excuse, me.

The Akka has it. We will get it?

akka {
  actor {
    serialize-messages = on
  }
}

• https://doc.akka.io/docs/akka/current/serialization.html#verification