sdk-core icon indicating copy to clipboard operation
sdk-core copied to clipboard

[Bug] Do not auto-retry gRPC-message-size-too-large errors

Open cretz opened this issue 2 years ago • 0 comments

Describe the bug

Not really a bug on this repo's part, but needs to be treated as one nonetheless.

When server gets > 4MB message, it returns ResourceExhausted error with something like:

grpc: received message larger than max (10248653 vs. 4194304)

Which is retried by our client as are all other ResourceExhausted. We need to stop retrying this one in particular but still retry other ResourceExhausted. If we can prevent it from going over the wire without manually pre-calculating the size that would be best, but otherwise we may need to hack something to change the error type for this particular error before it gets to the retry interceptor (or inside the retry interceptor if you can). Yes that may mean a message string check which is admittedly gross.

Also, add an integration test with a workflow whose activity payload will trigger this and confirm that it's a workflow task failure (can be in sdk-features).

cretz avatar Jan 05 '23 13:01 cretz