Introduce a unstable/chaos mode for ZetaClient

[lumtis]

Introduce a build for ZetaClient that simulate unstability.

Motivation

Localnet where we perform extended testing is a stable environment compared to live networks where you can have increased network usage or edge cases

Solution

Full implementation TBD but some ideas:

• Have an unstable boolean as part of the ZetaClient config, this variable is read in different locations in the client ect, to make some component failing. It can be updated dynamically

  • Advantage: I think this is the best solution because it would allow us to try it on live network with our own nodes for better performance testing

• Have it as a const in the codebase and using Linker Flags in builds to set it

  • Advantage: less flexible but "less risk" for observer to set it accidentally

• Define different build tags, and using different client implementation for client or middle-ware for failure simulation, the failing component are iserted at compilation time.

  • This might be a bit proper in the implement but this would be more complex

Example

This implementation is not clean but this is an example of simulated failure https://github.com/zeta-chain/node/pull/4186/commits/5609a6048a77f2867634a7235fd6b7b8aefcd822

[lumtis]

Better example for simulation of failure https://github.com/zeta-chain/node/pull/4191