RETRY() doesn't consult the Retry-After header if called with `quiet = TRUE`

[patr1ckm]

The API I work with sends a Retry-After header when rate limiting. httr::RETRY respects only retry-after (line). It seems that at least one definition of the protocol specifies Retry-After as the standard. Should RETRY automatically respect both cases by default?

[hadley]

Yeah, headers should be case insensitive.

[jennybc]

@patr1ckm Can you post a reproducible example of a request that fails due to this? Specifically, a live example of an API that returns a retry-after header with casing that is problematic?

I'm not able to reproduce this, largely because I don't know where to get a live-caught example of retry-after, in the first place, much less with specific capitalization. Every "fake" response I create, e.g. via httpbin, behaves as expected (httr::RETRY("GET", url = "http://httpbin.org/response-headers?reTRy-afteR=0.5")) w.r.t. headers.

Is there any chance that case sensitivity re: the retry-after header is not a problem after all and the only problem is that retry-after is not respected when quiet = TRUE?

[patr1ckm]

Is there any chance that case sensitivity re: the retry-after header is not a problem after all and the only problem is that retry-after is not respected when quiet = TRUE?

Yep! I'm pretty sure I was wrong about case sensitivity, and the only bug was not respecting the header if quiet = TRUE.

[jennybc]

OK here is the reprex. It's a bit fiddly to show because, of course, the wrong behaviour happens when quiet = TRUE but then that also makes RETRY() stop messaging about how long it's waiting!

faux_error <- function(h = NULL) {
  structure(list(status_code = 429, headers = h), class = "response")
}

# regular (truncated) exponential backoff,
testthat::with_mock(
  `httr::request_perform` = function(...) faux_error(),
  `httr::http_error` = function(...) TRUE,
  system.time(
    httr::RETRY("GET", "http://httpbin.org")  
  )
)
#> Request failed [429]. Retrying in 1.6 seconds...
#> Request failed [429]. Retrying in 2.7 seconds...
#>    user  system elapsed 
#>   0.035   0.003   4.334

# retry-after header present, quiet = FALSE (default)
# each wait time is 3 seconds, total wait time is 6 seconds
r <- faux_error(h = httr::insensitive(list(`reTRy-afteR` = 3)))
testthat::with_mock(
  `httr::request_perform` = function(...) r,
  `httr::http_error` = function(...) TRUE,
  system.time(
    httr::RETRY("GET", "http://httpbin.org")  
  )
)
#> Request failed [429]. Retrying in 3 seconds...
#> Request failed [429]. Retrying in 3 seconds...
#>    user  system elapsed 
#>   0.004   0.000   6.010

# retry-after header present, quiet = TRUE
# each wait time should be 3 seconds, total wait time should be 6 seconds
# but wait time is shorter, because we're doing regular exponential backoff
r <- faux_error(h = httr::insensitive(list(`reTRy-afteR` = 3)))
testthat::with_mock(
  `httr::request_perform` = function(...) r,
  `httr::http_error` = function(...) TRUE,
  system.time(
    httr::RETRY("GET", "http://httpbin.org", quiet = TRUE)  
  )
)
#>    user  system elapsed 
#>   0.003   0.000   2.599

^{Created on 2020-05-22 by the reprex package (v0.3.0.9001)}

[hadley]

httr has been superseded in favour of httr2, so is no longer under active development. If this problem is still important to you in httr2, I'd suggest filing an issue offer there 😄 — but I suspect that req_retry() already fixes this probelm. Thanks for using httr!