about uploading to FTP with curl

[cderv]

I was wondering if it is possible to upload a file to a FTP via curl package.

It is possible with curl with something like curl -T koc_dance.mp3 ftp://myftpsite.com/mp3/ --user myname:mypassword

With libcurl and its curl_easy_setopt, it is also possible.

I manage to do it for a text file creating a handle like this :

filetoupload <- file("file_to_upload.txt", "rb")

h <- new_handle() %>%
  handle_setopt(userpwd = "user:pwd", upload = 1L, readfunction = function(size) {
    readBin(filetoupload, raw(), size)
  })

However, after I am not sure wich function to use. I manage to upload the file with

con <- curl_fetch_memory("sftp://example/path/to/file_to_upload.txt", handle = h)

and checking that con$status_code == 0 but the return value form curl_fetch_memory is not so helpful. (no progress upload bar, or info that upload is 100%)

Should I use curl package like this ? Is there another solution ? Should this package include a curl_upload for ftp as it has a curl_download ? What is your opinion on this use case ?

Thanks

[jeroen]

Have a look at https://github.com/hadley/devtools/blob/master/R/upload-ftp.r

[cderv]

ok so it is hide in devtools for win-builder. It seems I got it right but spend some time on it whereas it already exists. thanks for the link. Why not include such a function in curl directly ?

[jeroen]

Yes youre right ill move this into curl. But its so simple you can just copy the code you need.

[cderv]

Yes I will do that. But it is not so easy to come with this code when you're not a curl expert. Seems simple now that I understand more about handles and curl options. It was a good learning experience to come with code. Maybe it could just be added as an example of usage for curl using handles and not included as a helper function.

[shrektan]

@jeroen Thanks for pointing out the code to upload to FTP. However, I find it's about 4~5 times slower than using a FTP software like Core FTP LE when uploading a large file.

It's tested in an internal network so I'm pretty sure it's not due to the traffic jam of the network...

Would you mind enlighten me the possible reason why using R's curl function is much slower? Is there any way to speed it up?

Moreover, curl::curl_download() the same file is quite fast, close to the speed in Core FTP LE...

My uploading code (cost 150s for a 800m file, very slow)

upload_ftp <- function(file, url, userpwd, verbose = FALSE) {
  stopifnot(file.exists(file))
  stopifnot(is.character(url))
  con <- file(file, open = "rb")
  on.exit(close(con))
  h <- curl::new_handle(upload = TRUE, filetime = FALSE, userpwd = userpwd)
  curl::handle_setopt(h, readfunction = function(n) {
    readBin(con, raw(), n = n)
  }, verbose = verbose)
  curl::curl_fetch_memory(url, handle = h)
}
system.time({
  upload_ftp(
    file = 'D:/data.rds', 
    url = 'sftp://xxx.xxx.xxx.xxx:22/ptf-info/data.rds',
    userpwd = "user:user",
    verbose = TRUE
  )
})

My downloading code (cost 40s for a 800m file, quite fast)

system.time(
  curl::curl_download(
    "sftp://xxx.xxx.xxx.xxx:22/ptf-info/data.rds", 
    destfile = 'D:/data0.rds',
    handle = new_handle(userpwd = "user:user")
  )
)