Mass destruction of R2DBC pools at once.

[pkgonan]

Mass destruction of R2DBC pools at once.

initial_size : 20 MAX_SIZE: 20 MAX_LIFE_TIME: 10 Seconds BACKGROUND_EVICTION_INTERVAL: 1 Seconds validation_depth: local validation_query: select 1

With the above configured, a show processlist; query on the database will automatically remove all connections without attempting to maintain 20 connections.

If we have a situation where 19 of the 20 connections have been open for 10 seconds and the remaining 1 connection has been open for 1 second, common sense would dictate that only 19 connections should be removed, but all 20 connections are removed.

Kotlin + Spring Boot Webflux + R2DBC + R2DBC Pool Configuration

@Configuration
internal class R2dbcConfiguration internal constructor(
    @Value("\${payment-platform.r2dbc.host}")
    private val host: String,
    @Value("\${payment-platform.r2dbc.port}")
    private val port: Int,
    @Value("\${payment-platform.r2dbc.username}")
    private val username: String,
    @Value("\${payment-platform.r2dbc.password}")
    private val password: String,
    @Value("\${payment-platform.r2dbc.database}")
    private val database: String,
): AbstractR2dbcConfiguration() {

    companion object {
        private const val CONNECTION_POOL_INITIAL_SIZE = 20
        private const val CONNECTION_POOL_MAX_SIZE = CONNECTION_POOL_INITIAL_SIZE
        private val CONNECTION_POOL_MAX_LIFE_TIME = Duration.ofSeconds(10)
        private const val VALIDATION_QUERY = "SELECT 1"
    }

    @Bean
    override fun connectionFactory(): ConnectionFactory {
        return ConnectionFactories.get(
            ConnectionFactoryOptions.builder()
                .option(ConnectionFactoryOptions.SSL, false)
                .option(ConnectionFactoryOptions.HOST, host)
                .option(ConnectionFactoryOptions.PORT, port)
                .option(ConnectionFactoryOptions.USER, username)
                .option(ConnectionFactoryOptions.PASSWORD, password)
                .option(ConnectionFactoryOptions.DATABASE, database)
                .option(ConnectionFactoryOptions.DRIVER, PoolingConnectionFactoryProvider.POOLING_DRIVER)
                .option(ConnectionFactoryOptions.PROTOCOL, MysqlConnectionFactoryProvider.MYSQL_DRIVER)
                .option(PoolingConnectionFactoryProvider.INITIAL_SIZE, CONNECTION_POOL_INITIAL_SIZE)
                .option(PoolingConnectionFactoryProvider.MAX_SIZE, CONNECTION_POOL_MAX_SIZE)
                .option(PoolingConnectionFactoryProvider.MAX_ACQUIRE_TIME, Duration.ofSeconds(2))
                .option(PoolingConnectionFactoryProvider.MAX_CREATE_CONNECTION_TIME, Duration.ofSeconds(2))
                .option(PoolingConnectionFactoryProvider.MAX_LIFE_TIME, CONNECTION_POOL_MAX_LIFE_TIME)
                .option(PoolingConnectionFactoryProvider.BACKGROUND_EVICTION_INTERVAL, Duration.ofSeconds(1))
                .option(PoolingConnectionFactoryProvider.VALIDATION_DEPTH, ValidationDepth.LOCAL)
                .option(PoolingConnectionFactoryProvider.VALIDATION_QUERY, VALIDATION_QUERY)
                .build()
        )
    }
}

[mp911de]

It sounds a bit as if this would originate from Reactor Pool as R2DBC pool delegates all eviction and statistics functionality to Reactor Pool. Paging @pderop for further guidance.

[pkgonan]

@mp911de @pderop

Hi, How do I prevent mass extinction of a Connection Pool?

[pderop]

@pkgonan, ok, I'll check this tomorrow morning.

[pkgonan]

Hi, @pderop Have you checked?

BACKGROUND_EVICTION_INTERVAL causes all connections to be removed and recreated.

However, all connections are recreated even if BACKGROUND_EVICTION_INTERVAL is not used.

[pderop]

I was in trouble on another PR, I'm now starting to check.

[pderop]

Can you confirm which exact version of reactor-pool you are using ?

I have tried to reproduce your scenario using the following junit test (based on 1.0.1 release), but I cannot reproduce the problem. Can you try to modify it, maybe I have missed your exact use case ? If you can reproduce it, please create an issue on reactor-pool project.

here is my junit test, to test it, you can load the reactor-pool (main branch) into your IDE and then add this test, for example in the SimpleDequePoolInstrumentationTest.

	@Test
	void useCase() throws InterruptedException {
		VirtualTimeScheduler vts = VirtualTimeScheduler.create();
		Duration maxLifeTime = Duration.ofSeconds(10);

		SimpleDequePool<String> pool = new SimpleDequePool<>(
				PoolBuilder
						.from(Mono.fromCallable(String::new))
						.clock(SchedulerClock.of(vts))
						.evictInBackground(Duration.ofMillis(1000), vts)
						.evictionPredicate((s, metadata) -> {
							return metadata.lifeTime() > maxLifeTime.toMillis();
						})
						.idleResourceReuseMruOrder()
						.sizeBetween(20, 20)
						.buildConfig());

		// Acquire 19 resources
		PooledRef<String>[] refs = IntStream.range(0, 19).mapToObj(i -> pool.acquire().block()).toArray(PooledRef[]::new);
		assertThat(pool.metrics().allocatedSize()).isEqualTo(20); // one more resource is created during lazy warmup
		assertThat(pool.metrics().idleSize()).isEqualTo(1);

		// wait for 1 sec
		vts.advanceTimeBy(Duration.ofMillis(1000));

		// acquire a twentieth resource (there is one remaining idle one)
		PooledRef<String> r20 = pool.acquire().block();
		// invalidate r20
		r20.invalidate().block();
		// re-acquire a new fresh r20 resource
		r20 = pool.acquire().block();

		// at this point, the first nineteen resources lifetime is around 1 sec, and r20 is fresh.
		assertThat(pool.metrics().allocatedSize()).isEqualTo(20);
		assertThat(pool.metrics().idleSize()).isEqualTo(0);

		// Now release all
		Stream.of(refs).forEach(ref -> ref.release().block());
		r20.release().block();

		// Wait 10 more sec, meaning that the first nineteen resources lifetime will be 11 sec
		vts.advanceTimeBy(Duration.ofMillis(10000));

		// the first nineteen resources should have been evicted, only 1 resource should still be acquired (r20)
		assertThat(pool.metrics().allocatedSize()).isEqualTo(1);

		// wait 1 sec, after that, r20 should have been evicted
		vts.advanceTimeBy(Duration.ofMillis(1000));
		assertThat(pool.metrics().allocatedSize()).isEqualTo(0);
	}