curl
curl copied to clipboard
curl
`-T -` always uploads with "Transfer-Encoding: chunked"
I did this
curl -T - localhost:40000 < ~/.bashrc
$ nc -l 40000
PUT / HTTP/1.1
Host: localhost:40000
User-Agent: curl/8.4.0
Accept: */*
Transfer-Encoding: chunked
Expect: 100-continue
30e
[...]
I expected the following
$ nc -l 40000
PUT / HTTP/1.1
Host: localhost:40000
User-Agent: curl/8.4.0
Accept: */*
Content-Length: 782
[...]
curl should realise that stdin is a regular file, and that it can do a non-chunked upload, like it would do if you used -T ~/.bashrc.
~~curl already does that when you use @- or <- with -F/-d and similar, but not when you use -T.~~ (EDIT: Actually, that is not true; I was misremembering; curl only knows the content length for stdin with -F/-d because it always loads the entire file into memory before sending it)
The problem is that curl -T never stats stdin to figure if it is a regular file when you use - as argument.
On Linux and BSDs, a possible workround is -T /dev/stdin (with --request-target /); since a path was provided, curl will stat it and realise it is a regular file, but it would be nicer if curl would just do that on its own.
I wanted to create a patch to fix this bug when I first learned about it, but the code that deals with stat-ing the file for upload (in src/tool_operate.c) is doing something special on #define __VMS systems to figure out the real upload size of regular files that involes opening the file, and reading all its data to count bytes. I didn't know why it did that, or how to do that for stdin if the system is VMS (maybe using lseek to go to the start of stdin), and anyway I would not be able to test the code I write for VMS on VMS, so I gave up trying to write the patch myself to avoid accidentally breaking the VMS port.
I remembered about this bug today, and noticed no one created an issue for it.
Ref: https://github.com/curl/curl/pull/11396#issuecomment-1613684489
curl/libcurl version
curl 8.4.0
curl 8.4.0 (x86_64-pc-linux-gnu) libcurl/8.4.0 OpenSSL/3.1.3 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.57.0
Release-Date: 2023-10-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zst
operating system
Arch Linux
Linux t420 6.1.57-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 11 Oct 2023 05:10:50 +0000 x86_64 GNU/Linux
I'm not following this.
curl uses chunked for -T from stdin because it does not know the size beforehand. How is that a bug?
It does not know its size only because it is not checking it, and there is no reason why it couldn't check it if stdin is a regular file.
If you run for example curl -T - localhost:40000 < ~/.bashrc, stdin is a file descriptor to a regular file, ~/.bashrc, so it could get its size using fstat() as it already does after open()ing a file when you do curl -T /path/to/file, and do an unchunked transfer, but it doesn't check and use the file size as content-length when you do -T -.
stdin is a file descriptor to a regular file
Wow, I had no idea it works like this...! :open_mouth: