Missing retry policy for ServerVersion.AutoDetect()

[lauxjpn]

trafficstars

There is currently no way to make use of the MySqlRetryingExecutionStrategy when it comes to ServerVersion.AutoDetect() calls, because ServerVersion.AutoDetect() is usually called before a DbContext has been created.

For example, if you run Pomelo on Windows Server prior to 2016, there exists a bug in the OS, that can lead to an exception about a too small buffer, when the SSL stream is decoded during the handshake while opening a MySQL connection (which is a known Windows bug that got fixed in Windows 10 and Windows Server 2016 at some point). This bug can appear pretty much randomly and just retrying to open the connection will work as a proper workaround.

Currently with Pomelo, the best way to handle retrying a ServerVersion.AutoDetect() call with Pomelo, is to do it manuall (or with Polly):

public class Context : DbContext
{
    private ServerVersion _serverVersion;

    public ServerVersion ServerVersion
        => _serverVersion ??= GetServerVersion();

    private ServerVersion GetServerVersion(string connectionString)
    {
        // Workaround Windows bug, that can lead to the following exception:
        //
        // MySqlConnector.MySqlException (0x80004005): SSL Authentication Error
        //     ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
        //     ---> System.ComponentModel.Win32Exception (0x8009030F): The message or signature supplied for verification has been altered
        //
        // See https://github.com/dotnet/runtime/issues/17005#issuecomment-305848835
        //
        // Also workaround for the fact, that ServerVersion.AutoDetect() does not use any retrying strategy.

#pragma warning disable EF1001
        var retryPolicy = Policy.Handle<Exception>(exception => MySqlTransientExceptionDetector.ShouldRetryOn(exception))
#pragma warning restore EF1001
            .WaitAndRetry(3, (count, context) => TimeSpan.FromMilliseconds(count * 250));

        var serverVersion = retryPolicy.Execute(() => serverVersion = ServerVersion.AutoDetect(connectionString));

        return serverVersion;
    }
}

[sgielen]

Just ran into this, I think. Before I had auto-detect version enabled, when my Docker container started up before the database, it would just keep reconnecting for a while until it connected properly, then continued with startup. Now, simply creating a DataContext fails:

                var dataContext = services.GetRequiredService<DataContext>();

With this exception:

  Unhandled exception. MySqlConnector.MySqlException (0x80004005): Unable to connect to any of the specified MySQL hosts.
     at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 433
     at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 363
     at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 94
     at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 124
     at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 915
     at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 406
     at MySqlConnector.MySqlConnection.Open() in /_/src/MySqlConnector/MySqlConnection.cs:line 369
     at Microsoft.EntityFrameworkCore.ServerVersion.AutoDetect(String connectionString)
     at ALeRTS.Backend.Host.Startup.<ConfigureServices>b__4_0(DbContextOptionsBuilder options) in /app/ALeRTS.Backend.Host/Startup.cs:line 53
     at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass1_0`2.<AddDbContext>b__0(IServiceProvider p, DbContextOptionsBuilder b)
     at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.CreateDbContextOptions[TContext](IServiceProvider applicationServiceProvider, Action`2 optionsAction)
     at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass17_0`1.<AddCoreServices>b__0(IServiceProvider p)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
     at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
     at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
     at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
     at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
     at Program.Main(String[] args) in /app/Program.cs:line 42

Any updates about this @lauxjpn ?

[lauxjpn]

@sgielen We have not yet decided, whether we are going to include a mechanism for this into Pomelo or not. People might have different needs and Polly already covers them sufficiently.

As for docker, use healthchecks to control when a container is considered started. See Control startup and shutdown order in Compose for more information.

[sgielen]

Thank you @lauxjpn! I didn't know about Compose condition, thank you for that, that helps too. In the meantime I have made my backend more resilient by trying to connect to the database multiple times during startup. Copying the code here in case it helps anyone else:

var host = CreateDefaultBuilder(.....).Build();

using (var scope = host.Services.CreateScope()) {
  var connectDeadline = DateTime.Now.AddMinutes(5);
  DataContext dataContext = null;
  while (dataContext == null) {
    try {
      dataContext = services.GetRequiredService<DataContext>();
    } catch(MySqlConnector.MySqlException e) {
      if (DateTime.Now > connectDeadline) {
        logger.LogError("Failed to connect to database, exiting.");
        throw;
      }
      logger.LogInformation("Failed to connect to database, retrying... {Error}", e);
      Thread.Sleep(5000);
    }
  }

  // do other things with dataContext if necessary
}