Capturing that retry attempts were exhausted

[samuelpordeus]

Hey, @wojtekmach! Thanks for the work on Req, most charming elixir HTTP lib ✨

I was wondering what's the best way to figure out that retry attempts were exhausted. Not sure if creating an issue is the best format for that, sorry if that's not the case 😅

We wanted to log something specific when a request fails after retries but I'm not sure if there's any way to do that with the current APIs

example:

iex(5)> TestApp.HttpClient.test_request(%{})
[error] retry: got exception, will retry in 1000ms, 3 attempts left
[error] ** (Mint.TransportError) connection refused
[error] retry: got exception, will retry in 2000ms, 2 attempts left
[error] ** (Mint.TransportError) connection refused
[error] retry: got exception, will retry in 4000ms, 1 attempt left
[error] ** (Mint.TransportError) connection refused
{:error,
 %{
   :__exception__ => true,
   :__struct__ => Mint.TransportError,
   :reason => :econnrefused
 }}

this error struct is exactly the same regardless of the retries. Is there any way to add something to it stating that retry attempts were all used?

If implementing something like this is a possibility for Req I could work on that.

[wojtekmach]

Thank you for the report. Creating issues is totally fine.

Unfortunately we cannot add anything to the %Mint.TransportError{} struct as we don't "own" it. However, for a while we were considering adding a %Req.Error{} struct that wraps the underlying exception. Importantly, it would have a :reason field that we could use in the retry step to check if we're supposed to retry it. It could have a :private field too and so any step could put arbitrary data there.

Before we do any of that, can you elaborate on your use case? If you have retries turned on and you still end up with an error then it is reasonable to assume retries were exhausted (otherwise it is a bug), why would you want to check it?

[samuelpordeus]

why would you want to check it?

when a request fails in this given application it usually means that a customer will end up creating a support ticket. so we want to create alerts for these error logs to act proactively, but as it is right now we might end up creating alerts even after successful retries.

[wojtekmach]

Would something like this work for you?

req = Req.new(...)

case Req.Request.execute(r) do
  {req, %Req.Response{} = resp} -> ...
  {req, exception} -> ...
end

in either case clause you'd be able to access req.private so you'd have access to all the information we keep around between requests. (There's no execute/1 function yet but I plan to add it.)

@heydtn you added response.private (thank you again!) and I'm curious whether API above would be compelling for your use cases? I think potentially so because you wouldn't have to write custom steps to access req.private and copy it to rest.private. Any feedback appreciated!

[wojtekmach]

I've added added Req.Request.request/1 which can be used like this:

req = Req.new(url: "https://httpbin.org/status/500")

Req.Request.request(req) do
  {:ok, request, response} ->
    IO.puts request.private.req_retry_count
    IO.puts "success: #{response.status}"

  {:error, request, exception} ->
    IO.inspect request.private.req_retry_count
    IO.puts "failure: " <> Exception.message(exception)
end