macOS: The Trust Settings Record was corrupted

[qwtel]

🐛 Bug Report

After upgrade to 1.12.2 I see the following when running wrangler dev:

thread 'tokio-runtime-worker' panicked at 'cannot access native cert store: Custom { kind: Other, error: Error { code: -25262, message: "The Trust Settings Record was corrupted." } }', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.21.0/src/connector.rs:46:30
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector>::new
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: tokio::runtime::task::core::Core<T,S>::poll
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   7: tokio::runtime::task::harness::Harness<T,S>::poll
   8: tokio::runtime::thread_pool::worker::Context::run_task
   9: tokio::runtime::thread_pool::worker::Context::run
  10: tokio::macros::scoped_tls::ScopedKey<T>::set
  11: tokio::runtime::thread_pool::worker::run
  12: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  13: tokio::runtime::task::core::Core<T,S>::poll
  14: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::blocking::pool::Inner::run
  17: tokio::runtime::context::enter
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Error: panic
   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new_unresolved
   2: failure::backtrace::internal::InternalBacktrace::new
   3: failure::backtrace::Backtrace::new
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
   6: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   7: std::thread::local::LocalKey<T>::with
   8: tokio::runtime::enter::Enter::block_on
   9: tokio::runtime::thread_pool::ThreadPool::block_on
  10: tokio::runtime::context::enter
  11: wrangler::commands::dev::edge::dev
  12: wrangler::commands::dev::dev
  13: wrangler::run
  14: wrangler::main
  15: std::sys_common::backtrace::__rust_begin_short_backtrace
  16: std::rt::lang_start::{{closure}}
  17: std::rt::lang_start_internal
  18: _main

Downgrade to 1.10.3 fixed the issue.

After tracking down the bug, I found similar issues:

• https://github.com/paritytech/polkadot/issues/1295
• https://github.com/ctz/rustls-native-certs/issues/4

What fixed the issue for 1.11.2 was deleting all "weird" certificates in KeyChain Access, but I couldn't track down which one specifically was causing it or how I've acquired it.

Environment

• operating system: macOS 11.0.1
• output of rustc -V: rustc 1.41.0 (5e1a79984 2020-01-27)
• output of node -v: v15.2.1
• output of wrangler -V: wrangler 1.12.2
• contents of wrangler.toml n/a

[biodranik]

Still reproduces on Mac OS X 10.15.7 with the latest wrangler 1.13.0

[xtuc]

it sounds like it's caused by a wrangler dependency, maybe upgrading it would fix the issue?

[biodranik]

Looks like the interfaces of some dependencies changed in newer versions, so it's not that easy (there are compile errors). I assume that bad versions are:

tokio-native-tls = "0.1.0"
tokio-rustls = "0.14.1"

In my case, manually deleting all wrong/expired certificates from the local keychain "solved" the problem.

[andrewdavidmackenzie]

Just installed latest wrangler:

❯ wrangler --version
👷 ✨  wrangler 1.13.0

on macos Big Sur 11.1

and I'm getting this error too:

❯ wrangler dev
💁  watching "./"
thread 'tokio-runtime-worker' panicked at 'cannot access native cert store: Custom { kind: Other, error: Error { code: -25262, message: "The Trust Settings Record was corrupted." } }', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-rustls-0.21.0/src/connector.rs:46:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: panic

The error message references /Users/runner but there is no such user or directory in my machine, not sure if that it the problem it's reporting or not...

[andrewdavidmackenzie]

I tracked it down to the "127.0.0.1" Certificate in "Keychain Access" "System" list. This had a valid date still (not expired) but the header area of the UI warned "This certificate is not trusted". I deleted it (requires root user password) and then wrangler dev starts fine.

It sort of makes sense, as it starts a web server on 127.0.0.1 .... but not sure of the details.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 180 days. It will be closed if no further activity occurs in the next week. Please feel free to comment if you'd like it to remain open, and thank you for your contributions.

[lostfocus]

Happens to me, too. Not sure which certificate it could be.

[hrz6976]

+1, facing this issue on macOS 10.15.7. The solution mentioned above is not working for me as cert for 127.0.0.1 is not showing up in the system keyring.

[hrz6976]

Digging into this issue, I found it was something with rustls-native-certs. It seems that this library simply raises an exception when parsing any root cert that can not be recognized. A suspicious root cert was located by:

1. clone https://github.com/rustls/rustls-native-certs (and install rust toolchain)
2. add a debug print line here

   for cert in iter {
        let der = cert.to_der();
   +    println!("Found cert {:?}", cert.subject_summary());
   

3. run cargo test

wrangler dev runs smoothly after removing the cert. buttom line: make sure all root certificates in the "Keychain Access" "System" entry is valid and trusted.

[remesta]

Solution by @12f23eddde worked like a charm! 🔥 With the additional println I was able to delete old certs in the system keyring. wrangle dev is working.

[mzagaja]

For future travelers after finding the certificate you may need to validate you have it with:

security find-certificate -c "name"

Then run to delete it:

security delete-certificate -c "name"

[mrbbot]

👋 Thanks for reporting this issue! Wrangler v1 is now deprecated and support is only being provided for critical updates or security concerns. As such, we are closing this issue.

New versions of Wrangler are maintained in the workers-sdk repo. If you are running into a similar issue with wrangler v2, please report it in the workers-sdk repo. For more info about wrangler v1 deprecation, please check out our blog post.

[varenc]

Just ran into this issue myself and this thread was the most helpful reference I found! For others, @12f23eddde instructions above were crucial for fixing it.

Turns out awhile ago I had manually marked a built-in CA root as untrusted. It didn't seem invalid, but this was enough to cause the error. Without @12f23eddde's advice it would have been a long time before I found it. Curiously after changing the cert back to "System Default", saving that setting, then changing it back again to "Never Trust", I didn't get rustls-native-certs error anymore. Hope this helps other struggling with this obscure issue!