Can't run cargo-audit in Cirrus CI with FreeBSD 13.3

[asomers]

In what looks like a repeat of #1058 , cargo-audit no longer works in Cirrus CI. It reports an error like this:

cargo audit
    Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
error: couldn't fetch advisory database: git operation failed: failed to fetch repo: An IO error occurred when talking to the server

I've tested the following program versions:

• 0.17.6 with FreeBSD 13.3 works
• 0.18.3 with FreeBSD 13.3 fails
• 0.18.3 with FreeBSD 13.2 works
• 0.19.0 with FreeBSD 13.3 fails
• 0.20.0 with FreeBSD 13.3 fails

This was all with various rustc nightly versions from within the last week. I don't know what's different about FreeBSD 13.3, nor do I understand the root cause of #1058 . But I will gladly help debug, if you can tell me what do look for.

[asomers]

Here's an expanded compatibility matrix:

	cargo-audit 0.17.6	cargo-audit 0.18.3	cargo-audit 0.19.0	cargo-audit 0.20.0
FreeBSD 13.2	✓	✓	✓	✓
FreeBSD 13.3	✓	x	x	x
FreeBSD 14.0	✓	x	x	x
FreeBSD 15.0-CURRENT	✓	x	x	x
Linux (rust:1.76.0)	✓	✓	✓	✓
OSX Sonoma	✓	✓	✓	✓

As you can see, cargo-audit 0.17.6 works everywhere. And all versions of cargo-audit work on FreeBSD 13.2. But versions 0.18.3 and later (which use gix instead of libgit2) fail on FreeBSD 13.3 and later. That's curious, because there aren't very many changes in FreeBSD 13.3. OpenSSL received only a few minor changes, mostly related documentation and performance. So it seems that there's a 3-way incompatibility between gix, FreeBSD 13.3, and something in Cirrus's environment.

	openssl
FreeBSD 13.2	1.1.1t
FreeBSD 13.3	1.1.1w
FreeBSD 14.0	3.0.12
FreeBSD 15.0-CURRENT	3.0.13
Linux (rust:1.76.0)	3.0.11
OSX Sonoma	3.2.1

https://www.freebsd.org/releases/13.3R/relnotes/ https://cirrus-ci.com/build/6549857133920256

[Shnatsel]

Thank you for the in-depth testing!

0.18.x and later should not be using OpenSSL at all, so the OpenSSL version is a non-factor.

I think we'll need to enable debug logging in reqwest to be able to diagnose this. I'm not sure if there is a built-in debug logging or if we'll need something like https://crates.io/crates/reqwest-tracing for that.

[asomers]

I don't see any way to enable tracing via an environment variable or something like that. But if you prepare a patch with the required level of tracing, then I can test it.

[dcoles]

I'm seeing a similar error in gitlab-cargo-audit calling rustsec::database::Database.fetch on Ubuntu 22.04.4 when rustsec crate is newer than 0.28.4:

rustsec-0.28.4:

$ RUST_LOG=reqwest=trace,hyper_util=trace cargo run --release
[2024-03-22T23:10:37Z TRACE reqwest::blocking::wait] (ThreadId(2)) park without timeout
[2024-03-22T23:10:37Z TRACE reqwest::blocking::client] (ThreadId(3)) start runtime::block_on
[2024-03-22T23:10:37Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:37Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999049s
[2024-03-22T23:10:37Z DEBUG reqwest::connect] starting new connection: https://github.com/
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999228s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999218s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999238s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999998207s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999049s
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] closing runtime thread (ThreadId(3))
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] signaled close for runtime thread (ThreadId(3))
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] (ThreadId(3)) Receiver is shutdown
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] (ThreadId(3)) end runtime::block_on
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] (ThreadId(3)) finished
[2024-03-22T23:10:38Z TRACE reqwest::blocking::client] closed runtime thread (ThreadId(3))

rustsec-0.28.5:

$ RUST_LOG=reqwest=trace,hyper_util=trace cargo run --release
[2024-03-22T23:11:36Z TRACE reqwest::blocking::wait] (ThreadId(2)) park without timeout
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] (ThreadId(3)) start runtime::block_on
[2024-03-22T23:11:36Z TRACE reqwest::blocking::wait] wait at most 30s
[2024-03-22T23:11:36Z TRACE reqwest::blocking::wait] (ThreadId(2)) park timeout 29.999999399s
[2024-03-22T23:11:36Z TRACE hyper_util::client::legacy::pool] checkout waiting for idle connection: ("https", github.com)
[2024-03-22T23:11:36Z DEBUG reqwest::connect] starting new connection: https://github.com/
[2024-03-22T23:11:36Z TRACE hyper_util::client::legacy::connect::http] Http::connect; scheme=Some("https"), host=Some("github.com"), port=None
[2024-03-22T23:11:36Z TRACE hyper_util::client::legacy::pool] checkout dropped for ("https", github.com)
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] closing runtime thread (ThreadId(3))
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] signaled close for runtime thread (ThreadId(3))
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] (ThreadId(3)) Receiver is shutdown
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] (ThreadId(3)) end runtime::block_on
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] (ThreadId(3)) finished
[2024-03-22T23:11:36Z TRACE reqwest::blocking::client] closed runtime thread (ThreadId(3))
Error: failed to fetch advisory-db

Caused by:
    git operation failed: failed to prepare fetch: An IO error occurred when talking to the server

Here are the respective dependencies:

$ cargo tree -p rustsec --depth 1
rustsec v0.28.4
├── cargo-lock v9.0.0
├── cvss v2.0.0
├── fs-err v2.11.0
│   [build-dependencies]
├── gix v0.55.2
├── home v0.5.9
├── platforms v3.3.0
├── semver v1.0.22
├── serde v1.0.197
├── tame-index v0.8.0
├── thiserror v1.0.58
├── time v0.3.34
├── toml v0.7.8
└── url v2.5.0

$ cargo tree -p rustsec --depth 1
rustsec v0.28.5
├── cargo-lock v9.0.0
├── cvss v2.0.0
├── fs-err v2.11.0
│   [build-dependencies]
├── gix v0.58.0
├── home v0.5.9
├── platforms v3.3.0
├── semver v1.0.22
├── serde v1.0.197
├── tame-index v0.9.7
├── thiserror v1.0.58
├── time v0.3.34
├── toml v0.7.8
└── url v2.5.0

I'm guessing likely related to either tame-index (v0.8.0→v0.9.7) or gix (v0.55.2→v0.58.0), since they seem to be the only direct dependency changes.

[asomers]

I've made progress:

TLDR;

FreeBSD users must install the ca_root_nss package in order to use cargo-audit.

Details

I was able to reproduce the failure locally (i.e. not in Cirrus-CI) on one FreeBSD 15.0 machine but not another. Running cargo-audit with ktrace showed that it tried to open may ssl-related files, including /usr/local/openssl/cert.pem. That file was present on the system where cargo-audit worked, but not on the system where it failed. The file comes from the ca_root_nss package. So I installed that, and now cargo-audit works. And I verified that installing ca_root_nss fixes the failures in Cirrus CI, too.

BUT, this problem only occurs if I install cargo-audit either from FreeBSD's package manager or from "cargo install". If instead I checkout the rustsec/rustec repo and checkout the cargo-audit/v0.19.0 tag, then run "cargo run --bin cargo-audit", it works. It works even if ca_root_nss is not present. ktrace shows that it's still trying to open the same file, but it doesn't care that it isn't there. Using the --release switch makes no difference.

What's different about building the tool with "cargo install" vs "cargo run"? "cargo install" will ignore the Cargo.lock file. Instead, it uses the latest versions of all dependencies. That suggests that maybe some dependency has had a regression. But I cannot reproduce the failure by doing a "cargo update" prior to "cargo run". Nor can I fix it by doing "cargo install --locked". So I can't explain the difference.

[Shnatsel]

BUT, this problem only occurs if I install cargo-audit either from FreeBSD's package manager or from "cargo install". If instead I checkout the rustsec/rustec repo and checkout the cargo-audit/v0.19.0 tag, then run "cargo run --bin cargo-audit", it works. It works even if ca_root_nss is not present.

This sounds like an issue with feature unification in the workspace. It's likely that rustsec-admin enables the webpki-roots package for the entire workspace when building form source. And when retrieving the local certificates fails, it falls back to the ones bundled with webpki-roots and ends up working.