curl or libcurl removes the encoding of white spaces (%20) on Windows

[salvafern]

Hi,

Apologies if this question has been raised before, I couldn't find a solution.

I am performing a GET request of an URL with encoded spaces. I noticed that curl - actually libcurl - is decoding %20, causing some requests fail. See this example:

term <- URLencode("Continental Shelf")
url <- paste0("http://marineregions.org/rest/getGazetteerRecordsByName.json/", term, "/true/false")

debug(curl::curl_fetch_memory)

request <- httr::GET(url)

httr::status_code(request)
#> [1] 400

This seems to be happening at curl_fetch_memory#L60, and more specifically either in R_curl_fetch_memory#8 or R_get_handle_response#493. See how the url loses the encoded space %20 after passing through curl_fetch_memory

# debugging in: curl::curl_fetch_memory(url, handle = handle)
# debug: {
#   nonblocking <- isTRUE(getOption("curl_interrupt", TRUE))
#   output <- .Call(R_curl_fetch_memory, enc2utf8(url), handle, 
#                   nonblocking)
#   res <- handle_data(handle)
#   res$content <- output
#   res
# }
Browse[6]> 
#   debug: nonblocking <- isTRUE(getOption("curl_interrupt", TRUE))
Browse[6]> url
# [1] "http://marineregions.org/rest/getGazetteerRecordsByName.json/Continental%20Shelf/true/false"
Browse[6]> 
#   debug: output <- .Call(R_curl_fetch_memory, enc2utf8(url), handle, nonblocking)
Browse[6]> 
#   debug: res <- handle_data(handle)
Browse[6]> 
#   debug: res$content <- output
Browse[6]> res$url
# [1] "https://marineregions.org/rest/getGazetteerRecordsByName.json/Continental Shelf/true/false"

For some reason, this happens only on windows, no problem in ubuntu.

R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

I am using these versions of libcurl and curl

libcurl/7.64.1 r-curl/4.3.2 

libcurl and C++ are a bit too far from my scope. Could you have a look and tell me what are we doing wrong? Is there any option of libcurl or curl that we can use to avoid this decoding?

Check this issue for my full sessionInfo() and further details (e.g. using '+' instead of spaces solves the problem)

This is affecting some packages that work with curl (via httr)

• https://github.com/ropensci/mregions/issues/60
• https://github.com/ropensci/worrms/issues/29
• https://github.com/ropensci/taxize/issues/888

Thanks in advance! Let me know if there is anything I can do. Salva