Windows Server 2016x64 and curl with Unicode support

[squromiv]

I did this

There is plenty of topics about Unicode issues in Windows. And here is another one. I try to use Windows Server 2016 and curl with so called Unicode support feature. So I have

Test string looks like

curl -v --data спасти http://httpbin.org/post

And this is the result

I expected the following

So what should I do in order to force this feature to work?

curl/libcurl version

8.1.1

operating system

Windows Server 2016 Standardx64

[jay]

How are you running the test string? I can't reproduce this from the command prompt. I don't have Windows Server 2016 but I tried Unicode builds of curl 8.1.1 in Windows 10 22H2 and Windows 7. I also tried a bunch of combinations including OpenSSL, schannel, and the latest curl. All sent 12 bytes (UTF-8 encoded спасти) and received the expected response (\u0441\u043f\u0430\u0441\u0442\u0438)

* !!! WARNING !!!
* This is a debug build of libcurl, do not use in production.
* STATE: INIT => CONNECT handle 0x290c468; line 1951 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => RESOLVING handle 0x290c468; line 1994 (connection #0)
* STATE: RESOLVING => CONNECTING handle 0x290c468; line 2068 (connection #0)
*   Trying 54.204.25.77:80...
* Connected to httpbin.org (54.204.25.77) port 80 (#0)
* STATE: CONNECTING => PROTOCONNECT handle 0x290c468; line 2112 (connection #0)
* STATE: PROTOCONNECT => DO handle 0x290c468; line 2142 (connection #0)
> POST /post HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/8.1.1-DEV
> Accept: */*
> Content-Length: 12
> Content-Type: application/x-www-form-urlencoded
>
* STATE: DO => DID handle 0x290c468; line 2236 (connection #0)
* STATE: DID => PERFORMING handle 0x290c468; line 2354 (connection #0)
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 200 OK
< Date: Wed, 01 Nov 2023 04:12:25 GMT
< Content-Type: application/json
< Content-Length: 457
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "\u0441\u043f\u0430\u0441\u0442\u0438": ""
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "12",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.1.1-DEV",
    "X-Amzn-Trace-Id": "Root=1-6541d029-22d1efc15d98347c5d24e8a0"
  },
  "json": null,
  "origin": "REMOVED-----",
  "url": "http://httpbin.org/post"
}
* STATE: PERFORMING => DONE handle 0x290c468; line 2553 (connection #0)
* multi_done: status: 0 prem: 0 done: 0
* Connection #0 to host httpbin.org left intact
* Expire cleared (transfer 0x290c468)

curl 8.1.1-DEV (i386-pc-win32) libcurl/8.1.1-DEV OpenSSL/3.0.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS Debug HSTS HTTPS-proxy IPv6 Largefile NTLM SSL Unicode UnixSockets

[squromiv]

How are you running the test string?

From cmd.exe.

I tried Unicode builds of curl 8.1.1

Could you please give a link to download exactly the same package of Curl, that you have used?

[jay]

Could you please give a link to download exactly the same package of Curl, that you have used?

I used custom built versions, however I just tried a CI build of CMake, VS2022, Debug, x64, Schannel, Static, Unicode (curl.exe direct or curl.zip copy) built from master 5f78cf50 and the received characters were UTF-8 as expected. You will need Visual Studio 2022 installed for ucrtbased.dll and vcruntime140d.dll dependencies, or download vs2022 debug runtime.zip for just the two files (both signed by MS). Put all the files in a temporary directory.

[squromiv]

Thanks a lot. Don`t know what is the reason, but this DEV build really works.

[jay]

Maybe there is a bug in your build procedure. How did you build curl?

[squromiv]

I used curl.exe from Anaconda package and was sure, that it is official build. Also I have downloaded the latest build from curl.se. And the command

curl.exe -V

shows, that it does not support Unicode feature at all. I'm at a loss. Is there any reliable source of compiled curl.exe with such a correctly working feature?

[jay]

I used curl.exe from Anaconda package and was sure, that it is official build.

It's not an official curl build. There are numerous Anaconda curl packages. Do you know which one?

I was able to reproduce with conda-forge curl curl-8.1.1-h68f0423_0 (and dependencies libcurl-8.1.1-h68f0423_0, libssh2-1.10.0-h680486a_3, openssl-1.1.1w-hcfcfb64_0, libzlib-1.2.13-hcfcfb64_5). curl with unicode problem.zip

They appear to be building curl using winbuild build method with ENABLE_UNICODE=yes and Visual Studio compiler so I don't see anything wrong with that.

Unicode arguments are always used if _UNICODE is defined which it is when the ENABLE_UNICODE build option is used.

https://github.com/curl/curl/blob/d755a5f7c009dd63a61b2c745180d8ba937cbfeb/winbuild/Makefile.vc#L119-L125

https://github.com/curl/curl/blob/d755a5f7c009dd63a61b2c745180d8ba937cbfeb/winbuild/MakefileBuild.vc#L532-L534

https://github.com/curl/curl/blob/d755a5f7c009dd63a61b2c745180d8ba937cbfeb/src/tool_main.c#L220-L230

@ocefpaf you are listed as a maintainer for curl-feedstock, do you have any idea what may be happening here?

Also I have downloaded the latest build from curl.se. And the command

curl.exe -V

shows, that it does not support Unicode feature at all.

The official curl build does not use Unicode because of some bugs.

[ocefpaf]

@ocefpaf you are listed as a maintainer for curl-feedstock, do you have any idea what may be happening here?

No idea at the moment but, b/c this seems to be a problem with our build, I recommend closing it here and opening an issue in the feedstock. We'll look into it ASAP.

[squromiv]

Do you know which one?

Sorry, but don`t know how to detect. It is located in this dir

C:\ProgramData\Anaconda3\Library\bin\curl.exe