SSL Authentication Error connecting to Amazon Aurora

[comagnan]

trafficstars

This happens randomly on the first call that opens a connection to the database with connection.Open(). The inner exception states "The message received was unexpected or badly formatted".

Here is the full stack trace.

MySql.Data.MySqlClient.MySqlException: SSL Authentication Error
 ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.
 ---> System.ComponentModel.Win32Exception: The message received was unexpected or badly formatted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at MySqlConnector.Core.ServerSession.<InitSslAsync>d__74.MoveNext()
   --- End of inner exception stack trace ---
   at MySqlConnector.Core.ServerSession.<InitSslAsync>d__74.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySqlConnector.Core.ServerSession.<ConnectAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySqlConnector.Core.ConnectionPool.<GetSessionAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at MySqlConnector.Core.ConnectionPool.<GetSessionAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.MySqlConnection.Open()

Over the hundreds of connections opened automatically during testing of MySqlConnector over the weekend, connections have failed to open roughly 90 times. In the majority of cases the library works as expected. When it fails, it's always on the first attempt at opening a connection since starting the process (then the process exits).

======== Some information about the setup:

MySqlConnector version: 0.46.0 Windows version: 10.0.14393.2430 .net version: 4.6.1 mySQL version: 5.6.10 (Aurora) Code is running inside of Docker containers (Enterprise Edition 18.03.1-ee-3)

[bgrainger]

Windows version: 10.0.14393.2430

Are you running on Windows 10 client or Windows Server?

Code is running inside of Docker containers (Enterprise Edition 18.03.1-ee-3)

Are these Windows Containers?

Would it be possible to get a packet capture of one of the failed connections?

[bgrainger]

This error has been previously reported with Aurora: #428

Also reported on Stack Overflow (also with Aurora); in this case the poster was using Connector/NET, but the System.Net.Security.SslState is extremely similar: https://stackoverflow.com/questions/46518563/net-mysql-error-a-call-to-sspi-failed-message-received-was-unexpected

The common factor here seems to be Amazon Aurora. The SO poster adds:

We do not receive this error in any other of our environments, which we are not running Aurora and simply just running MySQL 5.7.17 on Amazon RDS.

The evidence suggests a problem with Amazon Aurora, but I'm not sure how to “prove” that (or how to file a bug report for Amazon).

[comagnan]

The OS is Windows Server 2016, and these are Windows Containers. I'll see if I can get packet captures.

[jefraim]

@comagnan , were you able to resolve this issue?

Something similar also happens to our platform at random:

System.Data.Entity.Core.EntityException: The underlying provider failed on Open.
 ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.
 ---> System.ComponentModel.Win32Exception: The message received was unexpected or badly formatted
   --- End of inner exception stack trace ---
   at MySql.Data.Common.Ssl.StartSSL(Stream& baseStream, Encoding encoding, String connectionString)
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
   at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at System.Data.Common.DbConnection.OpenAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Data.Entity.Core.EntityClient.EntityConnection.<OpenAsync>d__8.MoveNext()
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.EntityConnection.<OpenAsync>d__8.MoveNext()

[bgrainger]

@jefraim Are you also using Amazon Aurora?

[jefraim]

@jefraim Are you also using Amazon Aurora?

We're using Amazon RDS MySQL,

[bgrainger]

@jefraim I just noticed that (from your call stack) you appear to be using Oracle's Connector/NET. This repo is for MySqlConnector, an OSS replacement for that library. To report a problem with that library, go to https://bugs.mysql.com/.

You could try switching to MySqlConnector https://www.nuget.org/packages/MySqlConnector/ if you like, but it's unlikely to help with this particular problem. It does fix a lot of other bugs, however: https://mysql-net.github.io/MySqlConnector/tutorials/migrating-from-connector-net/#fixed-bugs

[mikebollandajw]

We are also experiencing this with Amazon Aurora and Oracle Connector/Net

I will go to the relevent bug location to report this

https://bugs.mysql.com/bug.php?id=94240

[MikeAlhayek]

I am having the same problem. Here is my LINQ statement

Context.Integrations.Where(x => x.IsActive == isActive.Value).Include(x => x.Mappings)
                                                        .First();

I randomly get the following error

Error Message: Authentication failed, see inner exception.

Inner Error Message: The message or signature supplied for verification has been altered

Here is the stack trace

Stack Trace:    at MySql.Data.Common.Ssl.StartSSL(Stream& baseStream, Encoding encoding, String connectionString)
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
   at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ResultEnumerable`1.GetEnumerator()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider._TrackEntities[TOut,TIn](IEnumerable`1 results, QueryContext queryContext, IList`1 entityTrackingInfos, IList`1 entityAccessors)+MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass15_1`1.<CompileQueryCore>b__0(QueryContext qc)
   at System.Linq.Queryable.First[TSource](IQueryable`1 source)

[bgrainger]

@CrestApps You are using Connector/NET, created by Oracle. Their bug reporting site is https://bugs.mysql.com/.

Are you also using Amazon Aurora as the MySQL server? That appears to be the common factor in these bug reports.

[comagnan]

While I am unfortunately not in a position to give packet captures after all, I wanted to give some more information I found while comparing Connector/NET and MySqlConnector.

• SSL errors can be thrown by both connectors, as we can see above. The cause for the exception does appear to be tightly related to Aurora, as I was not able to reproduce it with a MySql docker container.
• There is however a difference in behavior between the two connectors (which is not necessarily a bug). Connector/NET does not wrap the AuthenticationException in a MySqlException, while MySqlConnector does.

These errors are temporary, so I ended up simply adding a retry on MySqlExceptions caused by AuthenticationException.

[mikebollandajw]

Does anyone have a link to the aurora/oracle ticket related to this?

[donaddon]

We are seeing the same issue connecting on first start of a new service process to an Amazon Aurora MySQL DB. Anyone have the link to the Oracle bug?

[bgrainger]

See above: https://github.com/mysql-net/MySqlConnector/issues/576#issuecomment-461378573

https://bugs.mysql.com/bug.php?id=94240

[kaplandani]

it happens also on mysql running on windows server (not related to amazon)

[AbrahamWilton]

it happens also on mysql running on windows server (not related to amazon)

In windows i have the same error, but resetting iis server . that's resolve my problem

[charlesgardiner]

I am experiencing this issue with AWS Aurora. Since this issue is 3 years old, I imagine a fix is not coming.

Has anyone found a work around when running against AWS Aurora?

[bgrainger]

It's a good idea to assume that the network is unreliable and that MySqlConnection.Open may fail for various reasons. So putting it in a retry loop (Polly is a common way) will handle this and other transient failures. That's the best "workaround" I can suggest right now.

[charlesgardiner]

@bgrainger Thanks, I appreciate your quick feedback.

[bgrainger]

All indications are that this is a problem on Aurora's end. There's not much MySqlConnector can do to work around it.

Recommendation is that clients retry opening a connection several times (e.g., with Polly) to recover from intermittent connection errors like this.