Issue with AWS Secrets Manager Rotation and Connection Handling in Webflux/Spring Cloud Gateway with r2dbc-mysql

[geunsik-finda]

Bug Report

Versions

• Driver: r2dbc-mysql:1.4.0
• Database: MySql 8.0.40
• Java: java 21
• OS: AWS Linux 2

issue

Subject: Issue with AWS Secrets Manager Rotation and Connection Handling in Webflux/Spring Cloud Gateway with r2dbc-mysql

We are encountering a peculiar issue in our Webflux-based Spring Cloud Gateway application (developed in Kotlin) that utilizes r2dbc for connecting to a MySQL database. Our setup involves:

Language: Kotlin Environment: Spring Cloud Gateway (with Webflux) Database: MySQL R2DBC Driver: r2dbc-mysql:1.4.0 Connection Pooling: r2dbc-pool:1.0.1 Secret Management: AWS Secrets Manager with rotation enabled.

The core problem arises during the rotation of the database credentials in AWS Secrets Manager. We have observed two distinct behaviors depending on our ConnectionFactoryOptions and ConnectionPoolConfiguration:

• Including ConnectionFactoryOptions.maxLifeTime: When we attempt to configure the connection factory options with .option(Option.valueOf("maxLifeTime"), Duration.ofMillis(poolMaxLifeTime)), the application does not seem to pick up the new credentials after AWS Secrets Manager performs a rotation. This leads to connection failures and requires a manual restart of the application to restore database connectivity.

• Excluding ConnectionFactoryOptions.maxLifeTime: If we exclude the maxLifeTime option from our ConnectionFactoryOptions, the credential rotation appears to be handled correctly at the connection factory level. However, in this scenario, we observe a significant increase in "Connection unexpectedly closed" log messages. This increased frequency of unexpected closures negatively impacts our monitoring and raises concerns about the stability and efficiency of the connections.

Interestingly, we are also configuring a maxLifeTime within the ConnectionPoolConfiguration:

@Bean
@Primary
fun connectionFactory(): ConnectionFactory {
    val secrets = secretsManagerConfiguration.getSecrets(username, DatabaseSecrets::class.java)
    val connectionFactoryOptionsBuilder = ConnectionFactoryOptions.builder()
        .option(ConnectionFactoryOptions.HOST, host)
        .option(ConnectionFactoryOptions.DATABASE, databaseName)
        .option(ConnectionFactoryOptions.DRIVER, "pool")
        .option(ConnectionFactoryOptions.PROTOCOL, r2dbcDriver)
        .option(MySqlConnectionFactoryProvider.CONNECTION_TIME_ZONE, "Asia/Seoul")
        .option(ConnectionFactoryOptions.USER, secrets.username)
        .option(ConnectionFactoryOptions.PASSWORD, secrets.password)
//        .option(Option.valueOf("maxLifeTime"), Duration.ofMillis(poolMaxLifeTime))    // this code work, but not work secrets manager rotate

    val connectionPoolConfiguration =
        ConnectionPoolConfiguration
            .builder(
                ConnectionFactories.get(connectionFactoryOptionsBuilder.build()),
            )
            .initialSize(poolInitialSize)
            .minIdle(poolMinIdle)
            .maxSize(poolMaxSize)
            .maxLifeTime(Duration.ofMillis(poolMaxLifeTime))
/*
We suspect that the maxLifeTime option only functions correctly when configured in ConnectionFactoryOptions in addition to ConnectionPoolConfiguration.
We have observed that setting it solely in ConnectionPoolConfiguration does not work the expected behavior.
*/
            .build()

    val connectionPool = ConnectionPool(connectionPoolConfiguration)
    connectionPool.warmup().subscribe()

    return connectionPool
}

We suspect that the maxLifeTime configuration, especially when applied at the ConnectionFactoryOptions level, might be interfering with the dynamic update of credentials during secret rotation. We are also unsure if the maxLifeTime configured in ConnectionPoolConfiguration is behaving as expected in this scenario.

We are seeking guidance on the following:

• Is this a known issue with the current versions of r2dbc-mysql and r2dbc-pool when used with AWS Secrets Manager rotation, particularly in a Kotlin/Webflux environment?
• Are there any specific configurations or strategies we should employ to handle secret rotation gracefully while ensuring connection stability and preventing the "Connection unexpectedly closed" errors?
• Could the maxLifeTime setting in either ConnectionFactoryOptions or ConnectionPoolConfiguration be contributing to these issues? If so, what are the recommended best practices for managing connection timeouts and lifetimes in a reactive, secret-rotated environment?
• Are there any plans to address these potential inconsistencies in future releases of r2dbc-pool or r2dbc-mysql?

Any insights or recommendations you can provide would be greatly appreciated. We are eager to find a robust solution for managing database credentials in our reactive application.

thank you.

[mp911de]

maxLifeTime requires a workaround in R2DBC Pool as it cannot be used together with max idle eviction. Since you've seen an issue, we configure an eviction predicate that decides whether to put a connection back into the pool. I suggest configuring backgroundEvictionInterval so that invalid connections can get evicted prior usage.

[jchrys]

hi, @geunsik-finda r2dbc-mysql has a password publisher option. https://github.com/asyncer-io/r2dbc-mysql/wiki/configuration-options#passwordpublisher