sysbench oltp performance of PostgreSQL drops after 30 minutes

[seybi87]

Hi sysbench community,

we are using sysbench to benchmark PostgreSQL 13.8 with oltp_read_write.lua workload and we target a total runtime of 90 minutes.

But we encounter a strange performance drop from 290 tps/s to 30 tps/s after 30 minutes and this behavior is reproducible, see the full log: 0_run.txt

We run the benchmarks on EC2 and PostgreSQL is deployed on a m5.large VM while sysbench is deployed on a c5.4xlarge VM. We monitor the system resources and there are not bottlenecks on the CPU, memory, disk or network level for PostgreSQL and sysbench.

PostgreSQL CPU utilization is around 40% before the performance drop.

We do not spot any issues on the system resource level or sysbench output that explains this performance drop.

The only unexpected behavior is detected in the PostgreSQL server.log (server.log) with a few duplicate key errors but would this explain the constant performance drop?

Find below also the full start commands:

Prepare:

/binary/sysbench/sysbench --report-interval=10  --threads=12 --time=5400 --table_size=1000000 --tables=100 --pgsql-host=172.31.17.133 --db-driver=pgsql --pgsql-user=benchant --pgsql-password=benchANT2022 --pgsql-db=benchdb /binary/sysbench/oltp_read_write.lua prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Run:

/binary/sysbench/sysbench --report-interval=10  --threads=50 --time=5400 --table_size=1000000 --tables=100 --pgsql-host=172.31.17.133 --db-driver=pgsql --pgsql-user=benchant --pgsql-password=benchANT2022 --pgsql-db=benchdb /binary/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Any advice on how to address this issue is much appreciated