dotnet
add libmsquic to to Helix queues with OpenSSL 1.1 (After 8/5/2022)
- [x] This issue is blocking
- [ ] This issue is causing unreasonable pain
This contributes to https://github.com/dotnet/runtime/issues/55639
So far, we have only Linux testing on Fedora x64. Since we are going to support HTTP/3 (that depends on Quic) in .NET 7, we should improve test coverage.
This was not done, because there are no sufficient packages of MsQuic as of now. MsQuic should cut 2.1 release for us by end of the week and we should have signed deb and rpm packages available on packages.microsoft.com shortly after.
The actually work consist of two steps:
-
adding packages.microsoft.com as source to given distributions. (unless already there) Example notes can be found here: https://docs.microsoft.com/en-us/dotnet/core/install/linux (we generally need to add the signing key and repo location)
-
use package manager to install MsQuic. For example 'apt install libmsquic'.
Note that MsQuic depends on OpenSSL 1.1 so we may have problems on distributions where OpenSSL 3.0 is the default. I think for now, we can skip them, active investigation into compatibility is in happening.
BTW there is currently no support of macOS packages and it will be addressed separately.
cc: @karelz
https://docs.microsoft.com/en-us/windows-server/administration/linux-package-repository-for-microsoft-software has probably better instructions how to add MS repo and how to consume packages published by MS.
Also note, that Ubuntu 22 has OpenSSL 3.0 by default. Msquic will need compat/legacy openssl-1.1 package. @rzikm verified that we an have both and all tests should pass.
Should be a fairly straightforward artifact to craft, I tried out the instructions and had no issues on Ubuntu 22.04 but the latest available was still 1.9.1
We will need 2.1 for 7.0. I'll update this thread once new version is available.
2.1 should be published now.
furt@ubu18:~$ sudo apt-get install libmsquic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
libmsquic
1 upgraded, 0 newly installed, 0 to remove and 102 not upgraded.
Need to get 6,402 kB of archives.
After this operation, 19.3 MB of additional disk space will be used.
Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 libmsquic amd64 2.1.0 [6,402 kB]
Fetched 6,402 kB in 1s (5,380 kB/s)
(Reading database ... 219616 files and directories currently installed.)
Preparing to unpack .../libmsquic_2.1.0_amd64.deb ...
Unpacking libmsquic (2.1.0) over (1.9.1) ...
Setting up libmsquic (2.1.0) ...
let me know if you have any troubles @MattGal
I verified this is in at least the following Helix images and all related internal/public versions of them
centos.7.amd64 redhat.7.amd64 sles.12.amd64 sles.15.amd64 ubuntu.1604.amd64 ubuntu.1804.amd64 ubuntu.2004.amd64 ubuntu.2204.amd64
Moving to rollout. @wfurt is trying to run the tests against the staging environment and we can do some experiments with this tomorrow if that doesn't work out.
Test run showed this didn't work... while the machines have the package, there seems to be indirect dependencies not expressed by libmsquic that prevent things from working.
@wfurt is investigating via our devtest lab and we'll work on finding these dependencies to see if that unblocks us.
The msquic package is not quite right. (I open issue for it)
toweinfu@toweinfu-u22:~$ ldd /usr/lib/x86_64-linux-gnu/libmsquic.so.2.1.0
linux-vdso.so.1 (0x00007ffd9b1fe000)
libcrypto.so.1.1 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd21fcd6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd21fcd1000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd21faa9000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd21fe32000)
for now the workaround is to
curl -O http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb
sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb
toweinfu@toweinfu-u22:/tmp$ ldd /usr/lib/x86_64-linux-gnu/libmsquic.so.2
linux-vdso.so.1 (0x00007ffc1c7af000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f31fc424000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f31fc41f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f31fc41a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f31fc1f2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f31fc851000)
little bot hacky but it seems like Ubuntu does not provide standard openss1.1 compatible package ;(
Redhat/Centos 7 has similar issue.
[toweinfu@toweinfu-r7 ~]$ ldd /usr/lib64/libmsquic.so.2
ldd: warning: you do not have execution permission for `/usr/lib64/libmsquic.so.2'
linux-vdso.so.1 => (0x00007ffc01bbb000)
libcrypto.so.1.1 => not found
libdl.so.2 => /lib64/libdl.so.2 (0x00007f1f79d23000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f79b07000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1f79739000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1f79f27000)
This should just work but it does not because of https://github.com/microsoft/msquic/issues/2975 The workaround is to add OpenSSL 1.1 explicitly:
sudo yum install -y openssl11
....
Installed:
openssl11.x86_64 1:1.1.1k-4.el7
[toweinfu@toweinfu-r7 ~]$ ldd /usr/lib64/libmsquic.so.2
ldd: warning: you do not have execution permission for `/usr/lib64/libmsquic.so.2'
linux-vdso.so.1 => (0x00007fffc22e1000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f97a63d9000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f97a61d5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f97a5fb9000)
libc.so.6 => /lib64/libc.so.6 (0x00007f97a5beb000)
libz.so.1 => /lib64/libz.so.1 (0x00007f97a59d5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f97a68bd000)
I did not find libmsquic on SLES.15
sudo rpm -Uvh https://packages.microsoft.com/config/sles/15/packages-microsoft-prod.rpm
sudo zypper install -y libmsquic
toweinfu@toweinfu-s15:~> ldd /usr/lib64/libmsquic.so.2.1.0
ldd: warning: you do not have execution permission for `/usr/lib64/libmsquic.so.2.1.0'
linux-vdso.so.1 (0x00007ffcc2cc3000)
libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007f841c311000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f841c10d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f841beea000)
libc.so.6 => /lib64/libc.so.6 (0x00007f841baf5000)
libz.so.1 => /lib64/libz.so.1 (0x00007f841b8de000)
/lib64/ld-linux-x86-64.so.2 (0x00007f841c7fa000)
ldd: warning: you do not have execution permission for `/usr/lib64/libmsquic.so.2.1.0'
toweinfu@toweinfu-s15:~> ls -al /usr/lib64/libmsquic.so*
-rw-r--r-- 1 root root 6652480 Aug 5 15:47 /usr/lib64/libmsquic.so.2
-rw-r--r-- 1 root root 6652480 Aug 5 15:47 /usr/lib64/libmsquic.so.2.1.0
that seems to be another problem with libmsquic package https://github.com/microsoft/msquic/issues/2977
The workaround would be to run
chmod 755 /usr/lib64/libmsquic*
It may not matter since we open libmsquic explicitly via dlopen but correcting permissions would be safe.
https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/24896 should address all these except SLES 15, this needs more investigation.
I got this running everywhere except Ubuntu 16.04 and Centos 7 (RHEL 7 is, confusingly, fine...):
// from https://engdata.kusto.windows.net
WorkItems
| where JobName in (
"6a1d3b91-f1ae-40a3-8725-0bb3f45d793b", // Centos 7 doesn't work
"fb4186a2-dded-4c9a-8ba7-20314b96acb0", // RHEL7 good
"fe8209d6-d1a3-4a47-b534-2a057ac3aae9", // SLES 12 good
"81f9f8ee-4058-453e-8f82-8d061f46521b", // SLES 15 good
"d00bf0eb-14d0-4f47-b59f-c5048a5f144d", // Ubuntu 16.04 doesn't work
"adcf6c0b-1044-4dde-ada2-5c6137d01b0f", // Ubuntu 18.04 good
"469631c7-3f8a-46f5-ae21-5bf43e44b060", // Ubuntu 20.04 is good
"963594b0-7a9a-4d04-97cc-6735a67e8e10") // Ubuntu 22.04 is good
| where Name != JobName
... @wfurt is taking a peek at the DevTest Lab
We've rolled out these changes and runtime does not need these tests to pass on Centos 7 directly (they use docker now) or ubuntu 16.04 (EOL OS / support on all the other versions). Closing as completed.
@MattGal We have found some more platforms which are missing MsQuic:
Mariner (docker image): https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9bdfb528372c45779a/System.Net.Quic.Functional.Tests/1/console.6d6097f9.log?helixlogtype=result OpenSuse 15.2: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-5f3cc5d8b53a49a396/System.Net.Quic.Functional.Tests/1/console.caebb575.log?helixlogtype=result (what is the difference between this and SLES?) CentOS 7: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75163-merge-aa86d5bd851f4aa8ae/System.Net.Quic.Functional.Tests/1/console.3ed81745.log?helixlogtype=result CentOS 8: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-30745a52006842c690/System.Net.Quic.Functional.Tests/1/console.4b717702.log?helixlogtype=result
there may be others, could you please have a look?
@rzikm all four of these examples are Docker images. Helix queues are VMs set up for direct execution of work items. ( You can tell from the using docker image at the top of the logs.)
LibMsquic can be added via simple PR to https://github.com/dotnet/dotnet-buildtools-prereqs-docker.
I'll handle trying to get these into the docker images at hand, but I hit issues on 4/4 of them:
- Mariner (docker image): Installs but all tests crash with
relocation error: /lib/libmsquic.so.2: symbol EVP_chacha20_poly1305 version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference - OpenSuse 15.2: Installs fine but 5 tests throw exceptions (rest pass; maybe an OK outcome?)
- CentOS 7: Installs fine, but the tests don't find libmsquic on system
- CentOS 8: There's no dotnet-buildtools-prereqs-dockerfiles version of this image.
It seems like OpenSuse runs on top of 16.04 -> that is VERY old. We should perhaps update the base to 20.04... We can take care of the containers. It seems like for pure RedHat we use VMs but for Centos we use containers, right?
There is new version of MsQuic as of today. I'll go through the containers to make sure we have updated msquic everywhere.
One more note, the Marriner may need extra binary build but I'm not sure what is the priority. cc: @nibanks
As @wfurt is working on this I've reassigned to him, happy to help if needed later though.
Centos 8 is EOL. Mariner seems low priority at the moment (and mat be hardest to fix) We (me & @rzikm should look at the rest)
As MsQuic 2.1.1 was just release, could you please rebuild current images so they pick up the changes @MattGal ?
I added msquic to OpenSUSE: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/690, the failing tests need to be disabled there. I have not have time to look at CentOS7 yet.
CentOS 7 has too old OpenSSL, so that is not going to work.
@wfurt I don't think there is more we need as part of this issue, so I am closing it again.