curl icon indicating copy to clipboard operation
curl copied to clipboard

curl_download URL encoding

Open ben-schwen opened this issue 4 years ago • 1 comments

As also mentioned here it seems that curl_download has a problem with url encoding of spaces etc.

library(curl)
url = "https://github.com/OpportunityInsights/EconomicTracker/raw/main/data/Google Mobility - County - Daily.csv"
tmpfile = tempfile()

# fails with HTTP error 400
curl_download(url, tmpfile)

# works
curl_download(utils::URLencode(url), tmpfile)

print(url)
print(utils::URLencode(url))

ben-schwen avatar Sep 15 '20 15:09 ben-schwen

Having the exact same problem, passing a url before encoding it leads to a 400 error. @jeroen should utils::URLencode() to correctly escape spaces before calling R_download_curl? At the moment I am relying on additional step in order to use curl:

utils::URLencode("https://www.mywebsite/badly encoded url.csv") %>%
    curl::curl_download(destfile='my_table.csv')

This obviously leads to code duplication and would probably better to bake url encoding directly into curl_download.

andodet avatar Jan 27 '21 08:01 andodet