Azure Connected Registry: Error "The SSL connection could not be established" in Nested Edge

[ksaye]

Describe the bug When using Connected Registry with Azure IoT Edge in a nested hierarchy with the IoTEdgeAPIProxy, a child Connected-Registry does not 'trust' the parent registry SSL cert.

To Reproduce Steps to reproduce the behavior:

1. Setup a nested Hierarchy for IoT Edge where the child (Level4) does not have access to the internet, but the parent (Level5) does
2. Successfully deploy the Connected-Registry to Level5
3. Attempt to deploy the Connected-Registry to Level4, which will fail

On Level4 you will see the following error:

[INFO] 2022-05-25T14:44:06Z [AcrOperationEvent] OperationName: ActivateAsync, CorrelationId: abf2dfb2-8ec3-41d6-b183-b96c26c83f99, Message: Attempting to activate connected registry 'ParentL4'
Startup of 'FrontEnd' failed with exception System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
  --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at ACR.OnPrem.FrontEnd.Services.ActivationService.InvokeActivateRequestAsync(String connectedRegistryName, String activationId, String correlationId, CancellationToken cancellationToken)
   at ACR.OnPrem.FrontEnd.Services.ActivationService.ActivateAsync(String activationFilePath, String activationId, CancellationToken cancellationToken)
   at ACR.OnPrem.FrontEnd.Services.ActivationService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at ACR.OnPrem.FrontEnd.Program.Main(String[] args)
[2022-05-25T14:44:07Z ERROR supervisor] process 'ACR.OnPrem.FrontEnd' has terminated, breaking out of monitor loop
[2022-05-25T14:44:07Z INFO  supervisor] shutting down all child processes
[2022-05-25T14:44:07Z WARN  supervisor] sending shutdown signal to 'registry serve /etc/acr/config.yml'
[2022-05-25T14:44:07Z INFO  supervisor] successfully sent a shutdown signal to 'registry serve /etc/acr/config.yml'
[2022-05-25T14:44:07Z INFO  supervisor] all child processes have shutdown
Error: exiting because a child process terminated

On Level 5, in the IoTEdgeAPIProxy log you will also see the error:

2022/05/25 14:44:06 [info] 1561#1561: *418 client closed connection while waiting for request, client: 192.168.15.92, server: 0.0.0.0:8000

Workaround Adding the following to the Create Options solves this:

{
    "HostConfig": {
        "Binds": [
            "/home/azureuser/connected-registry:/var/acr/data",
            "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates",
            "/etc/ssl/certs:/etc/ssl/certs"
        ]
    }
}

Expected behavior I would expect, like Level5, the Connected-Registry to come on line.

I believe that instead of mounting the /etc/ssl/certs and /usr/local/share/ca-certificates, Connected-Registry should call the IoTEdge and get the trust bundle.

Screenshots If applicable, add screenshots to help explain your problem.

Any relevant environment information

• OS: [e.g. Ubuntu, Windows] Ubuntu 18.04
• Azure CLI/PowerShell/SDK version
• Docker version: 20.10.16+azure-3
• Datetime (UTC) when the issue occurred: 5/25/2022
• Registry and image names

Additional context Add any other context about the problem here.

If any information is a concern to post here, you can create a support ticket or send an email to [email protected].