rdwd icon indicating copy to clipboard operation
rdwd copied to clipboard

use polite scraping

Open brry opened this issue 4 years ago • 1 comments

Consider using polite for downloading files. Maybe this is already conceptually fine with no re-downloads and CURL handle in https://github.com/brry/rdwd/blob/master/R/indexFTP.R#L121

brry avatar Jun 18 '20 08:06 brry

indexFTP might be fine, but dataDWD uses download.file with no checks. When calling dataDWD with, say, 500 urls, once banned, it will still keep trying the rest of the urls. Definitely not polite.

brry avatar Jun 18 '20 08:06 brry