rustls-ffi
rustls-ffi copied to clipboard
rustls
[Win32] Crashes with ASAN
I've built and used CrustTls in the demo and libcurl for some time with great success.
But enabling AddressSanitizer (ASAN) feature of latest MSVC-2019 (cl ver. 19.28.29912 for x86),
both curl https://whatever ... and a crustls-demo.exe www.vg.no / > test.file abort with this trace:
Sending:
GET / HTTP/1.1
Host: www.vg.no
...
ClientSession wants us to write_tls.
ClientSession wants us to read_tls. First we need to pull some bytes from the socket
=================================================================
==15612==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x07487200,0x07487735) and [0x0748727f, 0x074877b4) overlap
#0 0x611a1dcf (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x10031dcf)
#1 0x106beb in _ZN6rustls4msgs8deframer15MessageDeframer4read17hd804cc7f869bdd48E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x406beb)
#2 0x1129aa in _ZN74_$LT$rustls..client..ClientSession$u20$as$u20$rustls..session..Session$GT$8read_tls17h1cfd14ff7a451970E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x4129aa)
#3 0x102673 in rustls_client_session_read_tls (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x402673)
#4 0x24e247 in copy_tls_bytes_into_client_session F:\MingW32\src\inet\Crypto\Crustls\src\main.c:217
#5 0x24e5d3 in do_read F:\MingW32\src\inet\Crypto\Crustls\src\main.c:345
#6 0x24f621 in send_request_and_read_response F:\MingW32\src\inet\Crypto\Crustls\src\main.c:434
#7 0x24ebf0 in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
#8 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#9 0x7656fa28 (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
#10 0x771d7c7d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
#11 0x771d7c4d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)
0x07487200 is located 0 bytes inside of 18437-byte region [0x07487200,0x0748ba05) allocated by thread T0 here:
#0 0x611aa74c (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x1003a74c)
#1 0x140d8c in __rdl_alloc /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\alloc.rs:356
#2 0x112685 in _ZN6rustls6client13ClientSession3new17habc44068009b2037E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x412685)
#3 0x158a95 in _ZN7crustls6client25rustls_client_session_new28_$u7b$$u7b$closure$u7d$$u7d$17hfc2e02940a78157fE.llvm.16205319766425828457 (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x458a95)
#4 0x101b7d in rustls_client_session_new (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x401b7d)
#5 0x24ebbb in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
#6 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#7 0x7656fa28 (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
#8 0x771d7c7d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
#9 0x771d7c4d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)
0x0748727f is located 127 bytes inside of 18437-byte region [0x07487200,0x0748ba05) allocated by thread T0 here:
#0 0x611aa74c (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x1003a74c)
#1 0x140d8c in __rdl_alloc /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\alloc.rs:356
#2 0x112685 in _ZN6rustls6client13ClientSession3new17habc44068009b2037E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x412685)
#3 0x158a95 in _ZN7crustls6client25rustls_client_session_new28_$u7b$$u7b$closure$u7d$$u7d$17hfc2e02940a78157fE.llvm.16205319766425828457 (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x458a95)
#4 0x101b7d in rustls_client_session_new (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x401b7d)
#5 0x24ebbb in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
#6 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#7 0x7656fa28 (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
#8 0x771d7c7d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
#9 0x771d7c4d (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)
SUMMARY: AddressSanitizer: memcpy-param-overlap (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x10031dcf)
==15612==ABORTING
CFLAGS used: -fsanitize=address.
Ref: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes#16.9.0
A clang-cl v11 version with the same CFLAGS reports the same. The crash happens right after the 1st recv():
* 0.292 sec: src/main.c(319) (do_read+149):
recv (860, 0x0073E890, 2048, 0) --> 2048 bytes.
0000: 16 03 03 00 7A 02 00 00 76 03 03 46 77 60 1C 0A ....z...v..Fw`..
0010: 29 CA 39 FE 7B 37 6E 30 97 77 9C 69 F5 A5 51 2A )-9¦{7n0ùw£i)ÑQ*
0020: 8E B8 0B 64 CC A5 64 56 62 6F 15 20 54 3D 19 2B Ä+.d¦ÑdVbo. T=.+
0030: 5B 16 7C 0B C7 34 60 9A C0 88 A5 A8 D5 D6 92 71 [.|.¦4`Ü+êÑ¿++Æq
0040: A2 98 E2 13 2E 3D 57 33 30 7B FA 69 13 03 00 00 óÿG..=W30{·i....
0050: 2E 00 33 00 24 00 1D 00 20 80 AB 91 09 58 98 98 ..3.$... ǽæ.Xÿÿ
0060: 74 19 EB D8 97 98 EF 37 C5 88 80 E5 5A 72 17 10 t.d+ùÿn7+êÇsZr..
0070: 6C 52 23 7E 49 B1 7A ED 53 00 2B 00 02 03 04 14 lR#~I¦zfS.+.....
0080: 03 03 00 01 01 17 03 03 09 B7 C2 6D 9F 4B C1 92 .........+-mƒK-Æ
0090: 1E E6 5D BF 27 7F 44 C5 03 40 A8 E5 2C EC F8 0B .µ]+'.D+.@¿s,8°.
00A0: 3D CC 0C 13 27 11 6A 46 07 2A 40 2A 76 1F 17 36 =¦..'.jF.*@*v..6
00B0: BA 6A DD A5 2E 76 7D AA 08 1D C4 ED AF ED 87 C9 ¦j¦Ñ.v}¬..-f¤fç+
00C0: A6 01 5F 80 8D 6E 97 1B 33 7F EB D0 28 50 14 46 ª._Çìnù.3.d-(P.F
00D0: C8 90 73 1F 04 CF 05 9D 8A A1 5D 3F 3C 35 B1 7B +És..-.Øèí]?<5¦{
00E0: 86 69 C5 B0 40 E0 7E 01 FB B4 5B 0D 85 E7 5E 0A åi+¦@a~.v¦[.àt^.
00F0: 52 42 31 49 0B 0B E5 77 F5 C4 B0 55 0B E5 99 CA RB1I..sw)-¦U.sÖ-
0100: D5 44 3F D0 A9
(courtesy of my Wsock-trace library)
Thanks for the report. I've been able to reproduce, and get more detailed output by adding /DEBUG to the link command. I also added some printlns in the rustls code.
The 18437-byte region this error is referring to is the buf of a MessageDeframer inside the ClientSession: https://github.com/ctz/rustls/blob/v/0.19.0/rustls/src/msgs/deframer.rs#L10
The line that triggers the error is self.buf.copy_within:
https://github.com/ctz/rustls/blob/v/0.19.0/rustls/src/msgs/deframer.rs#L125-L126
In my case, this is called as copy_within(127..2048, 0). In other words, move the range starting at 127 to the beginning of the buf. Since the range is longer than 127 bytes, that means these are indeed overlapping, as the error says. However, this method shouldn't be using memcpy. The documentation explicitly says https://doc.rust-lang.org/std/primitive.slice.html#method.copy_within:
Copies elements from one part of the slice to another part of itself, using a memmove. src is the range within self to copy from. dest is the starting index of the range within self to copy to, which will have the same length as src. The two ranges may overlap. The ends of the two ranges must be less than or equal to self.len().
Next I'll try to generate a minimal repro case and see what I learn.
Here are the MS docs for this error: https://docs.microsoft.com/en-us/cpp/sanitizers/error-memcpy-param-overlap?view=msvc-160
Alright, here's a minimized repro:
copy_within.rs
fn do_it() {
let mut buf = [1, 2, 3];
buf.copy_within(1..3, 0);
assert_eq!(buf, [2, 3, 3]);
}
copy_within.c
int main() {
do_it()
}
rustc src\copy_within.rs --crate-type=staticlib --out-dir target\debug
cl -Focopy_within.obj -c src\copy_within.c -MD -fsanitize=address
link -out:copy_within.exe copy_within.obj target\debug\copy_within.lib userenv.lib ws2_32.lib advapi32.lib /DEBUG
copy_within.exe
Produces the AddressSanitizer: memcpy-param-overlap error. If I remove the -MD flag from cl, running copy_within.exe succeeds. Also if I replace it with -MT, copy_within.exe succeeds. Documentation for the MD flag is here: https://docs.microsoft.com/en-us/cpp/build/reference/md-mt-ld-use-run-time-library?view=msvc-160. It says:
All modules passed to a given invocation of the linker must have been compiled with the same run-time library compiler option (/MD, /MT, /LD).
/MD | Causes the application to use the multithread-specific and DLL-specific version of the run-time library. Defines _MT and _DLL and causes the compiler to place the library name MSVCRT.lib into the .obj file.
/MT | Causes the application to use the multithread, static version of the run-time library. Defines _MT and causes the compiler to place the library name LIBCMT.lib into the .obj file so that the linker will use LIBCMT.lib to resolve external symbols.
I'm guessing (but haven't confirmed and am not sure how to confirm) that Rust is building the .lib file with the moral equivalent of -MT, and so compiling the C code with -MD causes this issue. I'd definitely be curious to hear corroborating or competing theories.
more detailed output by adding /DEBUG to the link command.
AFAICS, that won't help for the .rscode. Not here anyway. Since those above symbols looks like
GNU-mangled symbols which the MSPDB engine does not understand.
Or is it possible to have Rust/Cargo generate .pdb files somehow? Perhaps I should have d/l a Rust built
by MSVC if such a thing exist. But I'm clueless when it comes to Rust.
If I remove the -MD flag from cl, running copy_within.exe succeeds. Also if I replace it with -MT, copy_within.exe succeeds.
The default for cl (if -MD is not specified) is -MT. I've no idea why it allows that since MicroSoft for some years now have been pushing for -MD and the UniversalCrt. Not sure if one can mix -MT (in CrustTls.lib) with -MD in e.g. libcurl.lib.
But it looks like you've found the problem. Good.
cl -Focopy_within.obj -c src\copy_within.c -MD -fsanitize=address
Try adding a -Zi and check if you get normal (non-GNU mangled) names in the report.
And BTW, with clang-cl, I also added -fsanitize-recover=address. But it did not help. It did not recover.
That flag do work; maybe not for this kind of error though.
I've found the problem, but I don't fully understand the problem. Maybe as a Windows dev you can help me understand this better.
To get more info, I also made a C version of this problem, where I had copy_within_lib.c, compiled to copy_within_lib.obj using /MT, and linked that against copy_within.obj, which contains main and was compiled with -fsanitize=address -MD. Sure enough, I got the same complaint from the address sanitizer. So the problem here is: you shouldn't mix /MD and /MT. If you do, unspecified bad things will happen. Maybe one of them is that the address sanitizer gives you spurious warnings? That seems like an oddly specific bad thing. @gvanem do you know what else is supposed to happen if you mix /MD and /MT?
Here's my current understanding of those flags: there are two main runtimes as part of MSVC: LIBCMT and MSVCRT. When you compile a C file using cl /MT, it inserts a linker directive that says "when you link this, link it against LIBCMT."
You can see that with dumpbin copy_within_lib.obj /directives and dumpbin copy_within.obj /directives. They show, respectively:
Linker Directives
-----------------
/DEFAULTLIB:LIBCMT
...
and
Linker Directives
-----------------
/DEFAULTLIB:LIBMSVCRT
...
And when I link those two, I do get a warning:
LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
However, the Rust copy_within.lib does not contain any such directive. Perhaps that's because it's a lib and not an obj? Anyhow when I link against the Rust lib, I don't get any warning - but I do get the same error from address sanitizer.
It seems like the solution, as much as there is one, is to be clear about whether things that link crustls should be built with /MD or /MT, and ideally produce an error if the wrong flag is used.
Here's some additional documentation, from the Rust side:
https://doc.rust-lang.org/reference/linkage.html#static-and-dynamic-c-runtimes
The standard library in general strives to support both statically linked and dynamically linked C runtimes for targets as appropriate. For example the x86_64-pc-windows-msvc and x86_64-unknown-linux-musl targets typically come with both runtimes and the user selects which one they'd like. All targets in the compiler have a default mode of linking to the C runtime. Typically targets are linked dynamically by default, but there are exceptions which are static by default such as: ... list of linux-only targets ...Crates may also learn about how the C runtime is being linked. Code on MSVC, for example, needs to be compiled differently (e.g. with /MT or /MD) depending on the runtime being linked. This is exported currently through the cfg attribute target_feature option:
However, whether I built the Rust library with +crt-static or -crt-static, the final linked binary generates the same address sanitizer error.
Any thoughts or elucidations you can provide would be much appreciated. Do you have thoughts on what, if anything, crustls should do here?
However, whether I built the Rust library with +crt-static or -crt-static, the final linked binary generates the same address sanitizer error.
A link -verbose could give more details. But I've just scraped the surface of Rust/Cargo; I've no idea really about which CRT
is used / expected with those flags.
Since rustc seems to be using MinGW (because of those awfully looking symbols), then using such a static MinGW library
in a MSVC program could be the root cause somehow.
Or it could simply be a false alarm in the ASAN libraries. To verify one would perhaps need to disassemble
the instructions around the memmove() calls to understand why ASAN bitches on this. The devil is in the details.
But I've not so much time to spend on this really. It was just when playing with the different SSL back-ends in libcurl
(set CURL_SSL_BACKEND=rustls) that I discovered this ASAN problem with rustls only. Thought you liked to know.
Thanks for the details. I suspect, given the newness of ASAN support in MSVC, this is more likely a false alarm in ASAN than a bug in crustls or rustls.
I've updated rustc / Cargo to version 1.51.0.
And updated my MSVC-2019. The the latest ASAN libraries are now dated 22-April.
Same exception with both cl and clang-cl.
But I've been experimenting with the env-var ASAN_OPTION (the list of them is huge! Try with a ASAN_OPTIONS=help=1).
Modifying my Makefile.Windows with:
test: all
export ASAN_OPTIONS=replace_intrin=0 ; \
crustls-demo.exe httpbin.org /headers
there is no memcpy-param-overlap exception.
But I'll also try with LLVM v12 released 9 days ago. To check if there was a false alarm that's fixed since v11.