sdk-core
sdk-core copied to clipboard
[Bug] Do not auto-retry gRPC-message-size-too-large errors
Describe the bug
Not really a bug on this repo's part, but needs to be treated as one nonetheless.
When server gets > 4MB message, it returns ResourceExhausted error with something like:
grpc: received message larger than max (10248653 vs. 4194304)
Which is retried by our client as are all other ResourceExhausted
. We need to stop retrying this one in particular but still retry other ResourceExhausted
. If we can prevent it from going over the wire without manually pre-calculating the size that would be best, but otherwise we may need to hack something to change the error type for this particular error before it gets to the retry interceptor (or inside the retry interceptor if you can). Yes that may mean a message string check which is admittedly gross.
Also, add an integration test with a workflow whose activity payload will trigger this and confirm that it's a workflow task failure (can be in sdk-features).