Timeouts catch you by surprise on mainnet: SDKs should _simulate_ timeouts during testing

[david-bakin-sl]

Problem

I'm always frustrated when ... code I write doesn't behave properly in the face of network timeouts because I a) forgot to write the timeout recovery code in the first place, and/or b) didn't test it (because it is difficult)

Solution

SDKs (all of ours) should allow fault-injection of timeouts using various schemes that the developer can configure. This would enable developers to easily test their timeout handlers and even remember to write them.

Different schemes could be like: a) Fail every nth call b) Fail every nth call of a specific API c) Randomly fail calls (given a percentage) d) Fail calls over an interval (e.g., fail 4 times then succeed) etc.

Deterministic choices would be useful in unit tests or when manually testing. Random choices would be useful when running stress tests, long-term tests, and such.

Alternatives

No response