Implement Fault Injection

[jamsajones]

[shubharao]

Would like to hear some use case details, especially about whether there is already an fault injection mechanism that you are using

[hbaldwi]

I'm interested in using this feature to verify that client and service mesh deadlines/retry policies are working correctly.

[bcelenza]

Hey @hbaldwi, cool idea. Can you tell us a little more about how you would test and verify the deadlines and retry policies are working? Is there a specific test framework and continuous integration pipeline you're targeting to make this work?

[hbaldwi]

As a naive implementation, I was planning to set a small percentage of the requests to be aborted with an error, and include an integration test for client services which would send a large number of unary RPCs (I'm using GRPC) to other services, and verify that there are no errors returned to the client. The retry policy would be set such that, if it was working correctly, the probability of an error being returned to the client would be so low that it would never realistically occur.
At the very least, this would verify that our client GRPC deadlines are sufficiently high for Envoy's retry policy (given its backoff strategy), and that the retry policy is functioning correctly. I'm using Jenkins for CI, and Golang's testing package for tests.

[kevinten10]

We plan to use chaos mesh for fault injection of services on appmesh