Retries don't work for HTTP POST resquests with a body

[aruiz14]

trafficstars

I'm using Tekton and I believe I've found a bug in this library, since it's used for sending CloudEvents based on Kubernetes events. By default, it uses the ExponentialBackoff retry strategy.

I modeled the sink to respond with a retriable HTTP status code, e.g. 425 Too Early, but I only observe 1 attempt. By inspecting the controller logs, I eventually see the give-up error message after 10 retries, which is something like:

Cloud Event Failure' Post ... : http: ContentLength=4137 with Body length 0 (10x)

I believe the reason for this is that the retry logic reuses the same http.Request object to perform multiple calls. The Body of the request is a io.ReaderCloser interface, but it's not "rewinded"/reset before trying to read again from it, so it ends up returning EOF. This is a known limitation of the standard library and there are libraries like https://github.com/hashicorp/go-retryablehttp that provide a solution for this exact problem.

[n3wscott]

oh interesting. Thanks for the report!

[embano1]

Fixed in #774