duplicati icon indicating copy to clipboard operation
duplicati copied to clipboard

Published 20 hours ago • verified publisher icon duplicati

"CERTIFICATE_VERIFY_FAILED" on all connection attempts

Open EKCJ opened this issue 2 years ago • 3 comments

  • [X] I have searched open and closed issues for duplicates.
  • [X] I have searched the forum for related topics.

Environment info

  • Duplicati version: 2.0.6.3_beta_2021-06-17
  • Operating system: UnRAID 6.9.2 2021-04-07 running the Duplicati docker image from https://hub.docker.com/r/duplicati/duplicati
  • Backend: Backblaze S2

Description

All backups, update checks and UsageReporter requests fail since 13/5/2022, giving the error "Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"

I have deleted and recreated the container (including deleting all local data - as part of trying to fix another issue). I have also run cert-sync as per this FAQ which reports that my cert list is current, and ca-mono-certificates appears to be the latest version (6.12.0.107) according to apt (which it should be anyway, as it is the version which is bundled with the docker image).

Other services running on the same host have internet access with no issues, there were no configuration changes prior to the issue occurring. I can also curl https URLs successfully from within the Duplicati container, although I don't know enough about SSL to know if that actually means anything.

Enabling the accept-any-ssl-certificate advanced option does not resolve the problem with the update checker, I haven't tried using it with a backup as I don't really want to transmit my backup insecurely. :)

Steps to reproduce

Run a backup, click "Check for updates" in the about page, or wait for a UsageReporter attempt

  • Actual result: Successful connection, backup completed or update check completed
  • Expected result: Backup or update check fails, error below

Debug log

Error when testing backup connection:

System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Backblaze.B2AuthHelper.get_Config () [0x0013d] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Backend.Backblaze.B2AuthHelper.get_APIUrl () [0x00000] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Backend.Backblaze.B2.List () [0x00011] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00000] in <fd3642a459884bd9a2412b4eda050109>:0 
  at Duplicati.Library.Backend.Backblaze.B2.Test () [0x00000] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00094] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <156011ea63b34859b4073abdbf0b1573>:0

Error when checking for updates:

System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
  at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate (Duplicati.Library.AutoUpdater.ReleaseType channel) [0x000ee] in <8d4cb1693e00483189d3952c3f0ed20f>:0

UsageReporter error:

{"ClassName":"System.Net.WebException","Message":"Error: TrustFailure (Authentication failed, see inner exception.)","Data":null,"InnerException":{"ClassName":"System.Security.Authentication.AuthenticationException","Message":"Authentication failed, see inner exception.","Data":null,"InnerException":{"Message":"Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED\n  at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132","Data":{},"InnerException":null,"StackTrace":"  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)\n  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 ","HelpLink":null,"Source":"System","HResult":-2146233088},"HelpURL":null,"StackTraceString":"  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 ","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233087,"Source":"mscorlib"},"HelpURL":null,"StackTraceString":"  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at System.Net.WebOperation.Run () [0x0009a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 \n  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 \n  at Duplicati.Library.Utility.AsyncHttpRequest.GetRequestStream (System.Int64 contentlength) [0x00068] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 \n  at Duplicati.Library.UsageReporter.ReportSetUploader+<>c.<Run>b__3_0 (CoCoL.IReadChannel`1[T] chan) [0x000c4] in <1e5a0a7e97af456e9e62533a54898b71>:0 ","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233079,"Source":"Duplicati.Library.Utility"}

EKCJ avatar May 16 '22 22:05 EKCJ

Is this on a Fedora system you recently updated to Fedora 36? If so, consider the following: https://github.com/duplicati/duplicati/issues/4650

DarkArc avatar May 20 '22 00:05 DarkArc

Thanks for this. I am running the Duplicati docker container which is based on Debian 10. For some reason, my backups have started working, although I'm still getting the same error on Updater and UsageReporter - I'm not too fussed about the updater at the moment, but if I start having problems with backups again I will look into the info in that issue.

EKCJ avatar May 21 '22 11:05 EKCJ

I'd have thought your backups would still fail, but I just wrote some information (much more in forum) at the following issue:

Failed to connect: Error: TrustFailure #3535

You should first check if you have the expired certificate in your mono store, and if so, remove it. These two topics might help.

TrustFailure error when using Backblaze B2

Error in updater - Trust Failure - SSL error

ts678 avatar May 21 '22 20:05 ts678

I was just checking my backups and discovered I've been hitting this issue for way too long.

@ts678 Wouldn't a new build of the duplicati/duplicati image fix it? Everything I'm seeing in those links involves updating a running container, which means I'd have to do that every time I need to recreate the duplicati container.

jerrac avatar Oct 01 '22 17:10 jerrac

Wouldn't a new build of the duplicati/duplicati image fix it?

I don't see how a simple rebuild of the image changes things, but I don't do Docker or its build. Do you have a theory in mind? Note that even getting a build out these days is difficult, due to lack of volunteers. Help with this project is very much needed... If someone wanted to try to figure out the Docker build, it might be possible to check in something to tweak certs during build, however I'm not sure how possible that is without running the image, and I'm not sure that happens. Maybe someone can see:

https://github.com/duplicati/duplicati/blob/master/Installer/Docker/build-images.sh

ts678 avatar Oct 03 '22 00:10 ts678

Presumably the new image would be based on an updated base image. If I'm reading the code right, that's the mono:6 image, and it had a release just 19 days ago. The certs on that image are likely up to date. Which would resolve this issue.

Of course I just checked the contents of the mono:6 and /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt is in there still. Deleting that file (thanks to a forum post I can't seem to find again) and running update-ca-certificates actually fixed the issue for me. So, hm. Maybe a new image wouldn't work.

In any event, I wrapped a custom Dockerfile around duplicati/duplicati to get my systems working again.

FROM duplicati/duplicati
RUN rm -f /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
RUN update-ca-certificates

I did look a little bit at building Duplicati myself. But I'm very much not a C# developer, and I run Linux for my dev work. A couple years ago I did try to get into it a bit deeper to try and work on the features I was asking for, but I didn't get very far.

Man, open source is awesome, but it sure can be tough at times. I'll poke at digging into Duplicati again, but I probably won't actually have time. :(

jerrac avatar Oct 03 '22 01:10 jerrac

https://github.com/mono/docker/blob/main/6.12.0.182/slim/Dockerfile may be what's in use. It says "FROM debian:buster-slim". Maybe being on Debian oldstable explains that older ca-certificates. The fixed version looks to be in bookworm - Debian testing. https://packages.debian.org/bookworm/ca-certificates shows it's on 20211016, and Changelog shows this change in 20211004:

  • Blacklist expired root certificate "DST Root CA X3" (closes: #995432)

ts678 avatar Oct 03 '22 02:10 ts678

custom Dockerfile

Would something like that in Duplicati's solve the issue? I don't think we can comfortably wait for mono to move to bookworm. If you like, feel free to do a pull request like that, although exactly how build pull requests get tested/proven is a mystery to me.

ts678 avatar Oct 04 '22 15:10 ts678

[Edit I managed to resolve my issue, info below my original comment]

I am having this issue as well, output below:

2023-01-19 10:43:27 +02 - [Warning-Duplicati.Library.Modules.Builtin.ReportHelper-ReportSubmitError]: Failed to send message: System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) --> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. --> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at /build/mono-6.12.0.182/external/boringssl/ssl/handshake_client.c:1132

This happens when trying to send a backup http or https report to duplicati-monitoring.com, changing the http report url from https to http doesn't solve the problem.

[My fix here.]

Remove DST_Root_CA_X3.crt from /usr/share/ca-certificates/mozilla/ Remove DST_Root_CA_X3.pem from /etc/ssl/certs/ run /usr/sbin/update-ca-certificates run /etc/ca-certificates/update.d/mono-keystore

This is on Debian11 with latest updates installed. The issue persisted after updates were done on Debian.

More discussion links:

https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157 https://community.synology.com/enu/forum/1/post/148065 https://forum.duplicati.com/t/unable-to-send-reports-to-duplicati-monitoring-com-due-to-ssl-issue/14623

Hope that helps anyone else happening on this error. Collaborators or mods are free to remove my comment if deemed irrelevant.

piqueza avatar Jan 19 '23 09:01 piqueza

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/plans-to-update-docker-image/16019/1

duplicatibot avatar Mar 29 '23 18:03 duplicatibot

@piqueza thank you SO much. I have been searching for a resolution to a Mono TrustFailure in a different Mono app and this is the only thing that resolved the issue on Ubuntu 16.04 Xenial (yes, I need to upgrade 😅)

theraccoonbear avatar May 12 '23 19:05 theraccoonbear

2.0.6.105_canary_2023-04-09 (or current Beta)

Remove obsolete Letsencrypt cert in Docker builds, thanks @Bubblesaway (forum)

remove obsolete Letsencrypt cert messing up certificate chaine #4918

should take care of

running the Duplicati docker image from https://hub.docker.com/r/duplicati/duplicati

which is almost as fixed as it can be, but possibly this issue should stay open longer so non-Docker users can find it more easily?

ts678 avatar Feb 01 '24 03:02 ts678