pgagroal icon indicating copy to clipboard operation
pgagroal copied to clipboard

WIP: Add new custom event loop for Linux I/O layer

Open hac-v opened this issue 1 year ago • 28 comments

Introduce ev.h and ev.c, establishing the foundation for the new custom event loop, pgagroal_ev.

Replace previous dependencies on libev with the custom event loop.

Implement support for io_uring with a fallback to epoll if io_uring is unavailable.

hac-v avatar Aug 14 '24 01:08 hac-v

@decarv Rebase, squash, and CI

jesperpedersen avatar Aug 18 '24 21:08 jesperpedersen

https://github.com/pgmoneta/pgmoneta/tree/main/test

jesperpedersen avatar Aug 21 '24 16:08 jesperpedersen

@decarv Please, look at the CI

jesperpedersen avatar Aug 24 '24 01:08 jesperpedersen

Please, see if you can get the liburing version check in place as well

jesperpedersen avatar Aug 24 '24 02:08 jesperpedersen

Run sudo mkdir -p /etc/pgagroal
kernel.io_uring_disabled = 0
pgagroal: PID file </tmp//pgagroal.2345.pid> exists, is there another instance running ?

Issue with shutdown ?

jesperpedersen avatar Sep 15 '24 20:09 jesperpedersen

pgagroal: PID file </tmp//pgagroal.2345.pid> exists, is there another instance running ?

Yes. I will investigate shutdown issues. However, from the CI output it could be anything, really.

hac-v avatar Sep 15 '24 20:09 hac-v

Try and a TRACE log file for gcc and clang, and see if there are pointers there...

jesperpedersen avatar Sep 15 '24 21:09 jesperpedersen

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/monitoring-and-troubleshooting-self-hosted-runners

jesperpedersen avatar Sep 15 '24 21:09 jesperpedersen

Please take a look at the latest commits. I decided to keep kqueue in this same PR for simplicity. I have just setup the testing environment for kqueue, so I haven't tested anything yet.

hac-v avatar Sep 22 '24 23:09 hac-v

Just a quick pass on the first part of the patch in order to give some comments that are probably general for the entire patch

jesperpedersen avatar Sep 23 '24 00:09 jesperpedersen

I still have issues with ipv6 listening addresses: pgagroal is unresponsive if listen_address = *. Changing listen_address to localhost allows me to test against pgbench, and so far it seems a little faster than the master branch, I'm still testing. pgagroal-cli shutdown works fine, while stopping with SIGINT causes the pid file to stale.

fluca1978 avatar Sep 23 '24 08:09 fluca1978

I still have issues with ipv6 listening addresses: pgagroal is unresponsive if listen_address = *. Changing listen_address to localhost allows me to test against pgbench, and so far it seems a little faster than the master branch, I'm still testing.

I will investigate that further to understand what is happening. I don't see how ev.c is doing this (although it has to be it).

pgagroal-cli shutdown works fine, while stopping with SIGINT causes the pid file to stale.

Thank you. Fixed. Already on the next commit.

hac-v avatar Sep 24 '24 17:09 hac-v

I still have issues with ipv6 listening addresses: pgagroal is unresponsive if listen_address = *. Changing listen_address to localhost allows me to test against pgbench, and so far it seems a little faster than the master branch, I'm still testing.

I will investigate that further to understand what is happening. I don't see how ev.c is doing this (although it has to be it).

@fluca1978 I think i have identified the issue. Can you please confirm you do not see this behavior in master branch? I also get an error in master branch... If you confirm that you can see a problem in master branch I will open another issue and fix it. Otherwise I will have to keep looking.

hac-v avatar Sep 30 '24 14:09 hac-v

@fluca1978 I think i have identified the issue. Can you please confirm you do not see this behavior in master branch? I also get an error in master branch... If you confirm that you can see a problem in master branch I will open another issue and fix it. Otherwise I will have to keep looking.

The problem with IPv6 addresses is present also in master, but on your branch the pgagrpal becomes unresponsive.

This is what happens with your branch having host = * (commit d5d66804d2329ee919f294280ab48357ab015b53:

% pgagroal                         
-> TRACE network.c:421 pgagroal_disconnect: fd=10
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:54322 (Invalid argument)
-> TRACE network.c:421 pgagroal_disconnect: fd=10
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:54322 (Invalid argument)
-> DEBUG ev.c:246 Available ev backends: io_uring, epoll
-> DEBUG ev.c:654 Event handling backend: 'io_uring'
-> TRACE network.c:421 pgagroal_disconnect: fd=15
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:8000 (Invalid argument)
-> TRACE network.c:421 pgagroal_disconnect: fd=15
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:8000 (Invalid argument)

and nothing happens, I'm unable to connect, to use pgagroal-cli and so on.

This is what happens with master (commit 40acff635d16e00eac08bf5307efdabc03ee13cd):

% pgagroal
pgagroal: Unknown key <ev_backend> with value <io_uring> in section [pgagroal] (line 80 of file </etc/pgagroal/pgagroal.conf>)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fee4:438d:54322 (Invalid argument)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fe80:c276:54322 (Invalid argument)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fee4:438d:8000 (Invalid argument)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fe80:c276:8000 (Invalid argument)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fee4:438d:6432 (Invalid argument)
-> DEBUG network.c:673 server: bind: fe80::a00:27ff:fe80:c276:6432 (Invalid argument)
-> INFO  main.c:1098 pgagroal: 2.0.0 started on *:54322
-> DEBUG main.c:1104 Socket: 5
-> DEBUG main.c:1104 Socket: 7
-> DEBUG main.c:1104 Socket: 8
-> DEBUG main.c:1104 Socket: 9
-> DEBUG main.c:1106 Unix Domain Socket: 4
-> DEBUG main.c:1107 Management: 3
-> DEBUG main.c:1110 Metrics: 12
-> DEBUG main.c:1110 Metrics: 13
-> DEBUG main.c:1110 Metrics: 14
-> DEBUG main.c:1110 Metrics: 15
-> DEBUG main.c:1114 Remote management: 16
-> DEBUG main.c:1114 Remote management: 17
-> DEBUG main.c:1114 Remote management: 18
-> DEBUG main.c:1114 Remote management: 19
-> DEBUG utils.c:417 libev available: select
-> DEBUG utils.c:421 libev available: poll
-> DEBUG utils.c:425 libev available: epoll
-> DEBUG utils.c:433 libev available: iouring
-> DEBUG main.c:1117 libev engine: epoll
-> DEBUG main.c:1118 Pipeline: 0
-> DEBUG main.c:1119 Pipeline size: 0
-> DEBUG main.c:1123 OpenSSL 3.0.7 1 Nov 2022
-> DEBUG main.c:1125 Configuration size: 874048
-> DEBUG main.c:1126 Max connections: 100
-> DEBUG main.c:1127 Known users: 3
-> DEBUG main.c:1128 Known frontend users: 1
-> DEBUG main.c:1129 Known admins: 1
-> DEBUG main.c:1130 Known superuser: No
-> TRACE server.c:75 pgagroal_get_primary: server (0) name (rachel) noninit_primary
-> DEBUG pool.c:967 pgagroal_prefill
-> TRACE server.c:75 pgagroal_get_primary: server (0) name (rachel) noninit_primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 0 fd 5
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> DEBUG security.c:591 Verify server mode: 0
-> DEBUG server.c:182 pgagroal_server_status:    #: 0
-> DEBUG server.c:183                         Name: rachel
-> DEBUG server.c:184                         Host: 127.0.0.1
-> DEBUG server.c:185                         Port: 5432
-> DEBUG server.c:195                         State: PRIMARY
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 0 FD 5
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 0 FD 21
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 0
-> TRACE server.c:64 pgagroal_get_primary: server (0) name (rachel) primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 1 fd 7
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 1 FD 7
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 1 FD 22
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 1
-> TRACE server.c:64 pgagroal_get_primary: server (0) name (rachel) primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 2 fd 8
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 2 FD 8
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 2 FD 23
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 2
-> TRACE server.c:64 pgagroal_get_primary: server (0) name (rachel) primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 3 fd 9
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 3 FD 9
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 3 FD 24
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 3
-> TRACE server.c:64 pgagroal_get_primary: server (0) name (rachel) primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 4 fd 12
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 4 FD 12
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 4 FD 25
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 4
-> TRACE server.c:64 pgagroal_get_primary: server (0) name (rachel) primary
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 5 fd 13
-> TRACE security.c:1227 Backend: R - Success
-> TRACE security.c:567 prefill_auth: auth type 0
-> TRACE security.c:2459 server_trust
-> TRACE security.c:596 prefill_auth: has_security 0
-> DEBUG security.c:597 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 5 FD 13
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1385 pgagroal: Management transfer connection: Slot 5 FD 26
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1401 pgagroal: Management return connection: Slot 5
-> DEBUG pool.c:1138 pgagroal_pool_status: 0/100
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 0
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Tue Oct  1 08:02:27 2024
-> DEBUG pool.c:1287                       Time: Tue Oct  1 08:02:27 2024
-> DEBUG pool.c:1288                       FD: 21
-> TRACE pool.c:1289                       PID: -1
-> TRACE pool.c:1290                       Auth: 0
-> TRACE pool.c:1293                       Size: 423

and so on.

Therefore the issues are:

  • a problem with IPv6, which is present in master too and has to be fixed outside of the uring work;
  • uring that becomes unresponsive with IPv6 or multiple interfaces.

Hope this helps.

fluca1978 avatar Oct 01 '24 07:10 fluca1978

@decarv See CI...

jesperpedersen avatar Oct 07 '24 07:10 jesperpedersen

@decarv See CI...

CI failed because PostgreSQL started on port 5433 instead of the default 5432, and our script specifically checks if PostgreSQL is running on port 5432. The service switched to port 5433 with no apparent reason. We could modify the script to check for subsequent ports, search for the PostgreSQL service dynamically, or try to prevent it from switching ports. I will just try to prevent PostgreSQL from switching ports, and, if the problem persists, I will adjust the CI to have the port being dinamically set.

hac-v avatar Oct 08 '24 18:10 hac-v

Where is :5433 coming from ? Restarting CI doesn't work

jesperpedersen avatar Oct 08 '24 21:10 jesperpedersen

CI need to work

jesperpedersen avatar Oct 08 '24 21:10 jesperpedersen

CI need to work

I am currently testing this on a personal branch and it is working. I haven't pushed it yet because I am adding back support for macos and it is currently failing. I will push without macos for now just so CI works.

hac-v avatar Oct 08 '24 22:10 hac-v

Please, take it out of draft, and PTAL us when it is ready

jesperpedersen avatar Oct 09 '24 03:10 jesperpedersen

@jesperpedersen @fluca1978 PTAL I’m currently testing for bugs, cleaning up old comments, and making mostly cosmetic changes. I’ll be updating the discussions forum as well.

hac-v avatar Oct 10 '24 22:10 hac-v

src/include/ev.h:56:10: fatal error: 'liburing.h' file not found
   56 | #include <liburing.h>
      |          ^~~~~~~~~~~~
1 error generated.

So, the necessary files aren't detected during the cmake phase. Of course the fix is to install liburing-devel...

jesperpedersen avatar Oct 11 '24 05:10 jesperpedersen

I'll look at the code later

jesperpedersen avatar Oct 11 '24 06:10 jesperpedersen

Testing version f60877ea0a2cf18566623f1f4e16dc31eec38225 with ev_backend = io_uring. The first run is fine, the second hangs (no restart or change of the configuration in between):

% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench
pgbench (16.4)
progress: 60.0 s, 1821.2 tps, lat 4.388 ms stddev 1.019, 0 failed
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 8
number of threads: 1
maximum number of tries: 1
duration: 60 s
number of transactions actually processed: 109279
number of failed transactions: 0 (0.000%)
latency average = 4.388 ms
latency stddev = 1.019 ms
initial connection time = 54.654 ms
tps = 1822.831288 (without initial connection time)
/usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T 60 -h localhost -p 54322 -U pgbenc  2.26s user 4.39s system 10% cpu 1:01.03 total


% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench
pgbench (16.4)
^C
/usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T 60 -h localhost -p 54322 -U pgbenc  0.00s user 0.01s system 0% cpu 11:25.60 total

I then CTL-C the server:

-> DEBUG worker.c:283 After client: PID 2522 Slot 9 (2)
-> WARN  management.c:1401 pgagroal_management_client_done: connect: 20
-> DEBUG worker.c:275 client disconnect: 19
-> DEBUG pool.c:1138 pgagroal_pool_status: 8/10
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 0
-> DEBUG pool.c:1271                       FD: 20
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 1
-> DEBUG pool.c:1271                       FD: 21
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 2
-> DEBUG pool.c:1271                       FD: 22
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 3
-> DEBUG pool.c:1271                       FD: 23
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 4
-> DEBUG pool.c:1271                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG worker.c:283 After client: PID 2524 Slot 8 (2)

but processes are still there:

%  ps -auxw | grep pgagroal 
luca        2317  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2318  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2319  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2320  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2321  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2322  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2324  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2326  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench

and in fact

% pgagroal
-> DEBUG network.c:675 server: bind: 127.0.0.1:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: 10.0.2.15:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: 192.168.222.50:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: ::1:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:54322 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:54322 (Invalid argument)
-> FATAL main.c:922 pgagroal: Could not bind to *:54322

fluca1978 avatar Oct 11 '24 09:10 fluca1978

Testing version f60877e with ev_backend = io_uring. The first run is fine, the second hangs (no restart or change of the configuration in between):

% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench
pgbench (16.4)
progress: 60.0 s, 1821.2 tps, lat 4.388 ms stddev 1.019, 0 failed
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 8
number of threads: 1
maximum number of tries: 1
duration: 60 s
number of transactions actually processed: 109279
number of failed transactions: 0 (0.000%)
latency average = 4.388 ms
latency stddev = 1.019 ms
initial connection time = 54.654 ms
tps = 1822.831288 (without initial connection time)
/usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T 60 -h localhost -p 54322 -U pgbenc  2.26s user 4.39s system 10% cpu 1:01.03 total


% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench
pgbench (16.4)
^C
/usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T 60 -h localhost -p 54322 -U pgbenc  0.00s user 0.01s system 0% cpu 11:25.60 total

I then CTL-C the server:

-> DEBUG worker.c:283 After client: PID 2522 Slot 9 (2)
-> WARN  management.c:1401 pgagroal_management_client_done: connect: 20
-> DEBUG worker.c:275 client disconnect: 19
-> DEBUG pool.c:1138 pgagroal_pool_status: 8/10
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 0
-> DEBUG pool.c:1271                       FD: 20
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 1
-> DEBUG pool.c:1271                       FD: 21
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 2
-> DEBUG pool.c:1271                       FD: 22
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 3
-> DEBUG pool.c:1271                       FD: 23
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 4
-> DEBUG pool.c:1271                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG worker.c:283 After client: PID 2524 Slot 8 (2)

but processes are still there:

%  ps -auxw | grep pgagroal 
luca        2317  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2318  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2319  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2320  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2321  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2322  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2324  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench
luca        2326  0.0  0.2  13716  2768 pts/0    S    10:59   0:01 pgagroal: pgbench/pgbench

and in fact

% pgagroal
-> DEBUG network.c:675 server: bind: 127.0.0.1:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: 10.0.2.15:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: 192.168.222.50:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: ::1:54322 (Address already in use)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:54322 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:54322 (Invalid argument)
-> FATAL main.c:922 pgagroal: Could not bind to *:54322

Thanks for the report, Luca. I believe this was fixed. Can you please confirm?

hac-v avatar Oct 17 '24 22:10 hac-v

Thanks for the report, Luca. I believe this was fixed. Can you please confirm?

Uhm, I'm getting the very same behavior on 4df7ed63d4b7f515195acb33f957280778ea9320 (please note that the status of the connections is reported as replica, but this has been fixed, I think there is the need for a rebase).

% pgagroal
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:54322 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:54322 (Invalid argument)
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:8000 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:8000 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fee4:438d:6432 (Invalid argument)
-> DEBUG network.c:675 server: bind: fe80::a00:27ff:fe80:c276:6432 (Invalid argument)
-> INFO  main.c:1094 pgagroal: 2.0.0 started on *:54322
-> DEBUG main.c:1100 Socket: 5
-> DEBUG main.c:1100 Socket: 7
-> DEBUG main.c:1100 Socket: 8
-> DEBUG main.c:1100 Socket: 9
-> DEBUG main.c:1102 Unix Domain Socket: 4
-> DEBUG main.c:1103 Management: 3
-> DEBUG main.c:1106 Metrics: 11
-> DEBUG main.c:1106 Metrics: 12
-> DEBUG main.c:1106 Metrics: 13
-> DEBUG main.c:1106 Metrics: 14
-> DEBUG main.c:1110 Remote management: 15
-> DEBUG main.c:1110 Remote management: 16
-> DEBUG main.c:1110 Remote management: 17
-> DEBUG main.c:1110 Remote management: 18
-> DEBUG main.c:1113 Pipeline: 0
-> DEBUG main.c:1114 Pipeline size: 0
-> DEBUG main.c:1118 OpenSSL 3.0.7 1 Nov 2022
-> DEBUG main.c:1120 Configuration size: 361408
-> DEBUG main.c:1121 Max connections: 10
-> DEBUG main.c:1122 Known users: 3
-> DEBUG main.c:1123 Known frontend users: 1
-> DEBUG main.c:1124 Known admins: 1
-> DEBUG main.c:1125 Known superuser: No
-> DEBUG pool.c:967 pgagroal_prefill
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 0 fd 5
-> DEBUG security.c:592 Verify server mode: 0
-> DEBUG server.c:182 pgagroal_server_status:    #: 0
-> DEBUG server.c:183                         Name: rachel
-> DEBUG server.c:184                         Host: 127.0.0.1
-> DEBUG server.c:185                         Port: 5432
-> DEBUG server.c:195                         State: PRIMARY
-> DEBUG security.c:598 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 0 FD 5
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 0 FD 20
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 0
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 1 fd 7
-> DEBUG security.c:598 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 1 FD 7
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 1 FD 21
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 1
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 2 fd 8
-> DEBUG security.c:598 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 2 FD 8
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 2 FD 22
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 2
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 3 fd 9
-> DEBUG security.c:598 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 3 FD 9
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 3 FD 23
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 3
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 4 fd 11
-> DEBUG security.c:598 prefill_auth: SUCCESS
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 4 FD 11
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 4 FD 24
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 4
-> DEBUG pool.c:1138 pgagroal_pool_status: 0/10
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 0
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 20
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 1
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 21
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 2
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1














# in the meantime in another terminal
% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench






-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 0
-> DEBUG security.c:405 authenticate: got pooled connection (0)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 0 (19 -> 20)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 1/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 1
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 21
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 2
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 0 FD 20
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 0
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1502 pgagroal: Management client done
-> DEBUG worker.c:275 client disconnect: 19
-> DEBUG pool.c:1138 pgagroal_pool_status: 0/10
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 0
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1288                       FD: 20
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 1
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 21
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 2
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG worker.c:283 After client: PID 267874 Slot 0 (1)
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 0
-> DEBUG security.c:405 authenticate: got pooled connection (0)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 0 (19 -> 20)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 1/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 1
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 21
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 2
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 1
-> DEBUG security.c:405 authenticate: got pooled connection (1)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 1 (19 -> 21)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 2/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 2
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 2
-> DEBUG security.c:405 authenticate: got pooled connection (2)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 2 (19 -> 22)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 3/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 3
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 3
-> DEBUG security.c:405 authenticate: got pooled connection (3)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 3 (19 -> 23)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 4/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 4
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1288                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 4
-> DEBUG security.c:405 authenticate: got pooled connection (4)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 4 (19 -> 24)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 5/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 5
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 5 fd 25
-> DEBUG security.c:409 authenticate: creating pooled connection
-> DEBUG security.c:1673 client_trust 19 5
-> DEBUG security.c:421 authenticate: created pooled connection (5)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 5 (19 -> 25)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 6/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 6
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 6 fd 25
-> DEBUG security.c:409 authenticate: creating pooled connection
-> DEBUG security.c:1673 client_trust 19 6
-> DEBUG security.c:421 authenticate: created pooled connection (6)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 6 (19 -> 25)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 7/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 7
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 7 fd 25
-> DEBUG security.c:409 authenticate: creating pooled connection
-> DEBUG security.c:1673 client_trust 19 7
-> DEBUG security.c:421 authenticate: created pooled connection (7)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 7 (19 -> 25)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 8/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 7
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 8
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 2 FD 22
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 6 FD 25
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 0 FD 20
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 1 FD 21
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 5 FD 25
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 3 FD 23
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 7 FD 25
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 4 FD 24












# and now from another terminal
% time /usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T  60 -h localhost -p 54322 -U pgbench pgbench


-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 8 fd 25
-> DEBUG security.c:409 authenticate: creating pooled connection
-> DEBUG security.c:1673 client_trust 19 8
-> DEBUG security.c:421 authenticate: created pooled connection (8)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 8 (19 -> 25)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 9/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 7
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 8
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> INFO  worker.c:204 disconnect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:425 pgagroal_return_connection: Slot 8 FD 25
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1370 pgagroal: Management transfer connection: Slot 8 FD 25
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1386 pgagroal: Management return connection: Slot 8
-> DEBUG management.c:1852 pgagroal-cli version 20000
-> DEBUG main.c:1502 pgagroal: Management client done
-> DEBUG worker.c:275 client disconnect: 19
-> DEBUG pool.c:1138 pgagroal_pool_status: 8/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 7
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1279 pgagroal_pool_status: State: FREE
-> DEBUG pool.c:1280                       Slot: 8
-> DEBUG pool.c:1281                       Server: 0
-> DEBUG pool.c:1282                       User: pgbench
-> DEBUG pool.c:1283                       Database: pgbench
-> DEBUG pool.c:1284                       AppName: 
-> DEBUG pool.c:1285                       Rule: 0
-> DEBUG pool.c:1286                       Start: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1287                       Time: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1288                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG worker.c:283 After client: PID 267894 Slot 8 (1)
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG security.c:392 authenticate: getting pooled connection
-> DEBUG security.c:1673 client_trust 19 8
-> DEBUG security.c:405 authenticate: got pooled connection (8)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 8 (19 -> 25)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 9/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 7
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 8
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1269 pgagroal_pool_status: State: NOTINIT
-> DEBUG pool.c:1270                       Slot: 9
-> DEBUG pool.c:1271                       FD: -1
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19
-> DEBUG pool.c:172 connect: server 0
-> DEBUG pool.c:211 connect: 127.0.0.1:5432 using slot 9 fd 26
-> DEBUG security.c:409 authenticate: creating pooled connection
-> DEBUG security.c:1673 client_trust 19 9
-> DEBUG security.c:421 authenticate: created pooled connection (9)
-> DEBUG security.c:431 authenticate: SUCCESS
-> DEBUG worker.c:95 pgagroal_worker: Slot 9 (19 -> 26)
-> INFO  worker.c:101 connect: user=pgbench database=pgbench address=::ffff:ffff:ffff:ffff
-> DEBUG pool.c:1138 pgagroal_pool_status: 10/10
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 0
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 20
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 1
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 21
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 2
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 22
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 3
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 23
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 4
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:47 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 24
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 5
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 6
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 7
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:11:52 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 8
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1309                       FD: 25
-> DEBUG pool.c:1300 pgagroal_pool_status: State: IN_USE
-> DEBUG pool.c:1301                       Slot: 9
-> DEBUG pool.c:1302                       Server: 0
-> DEBUG pool.c:1303                       User: pgbench
-> DEBUG pool.c:1304                       Database: pgbench
-> DEBUG pool.c:1305                       AppName: pgbench
-> DEBUG pool.c:1306                       Rule: 0
-> DEBUG pool.c:1307                       Start: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1308                       Time: Fri Oct 18 08:14:09 2024
-> DEBUG pool.c:1309                       FD: 26
-> DEBUG ev.c:582 Available ev backends: io_uring, epoll
-> DEBUG ev.c:597 Selected backend: 'io_uring'
-> DEBUG security.c:240 SSL request from client: 19


# here it hangs ...
^C
/usr/pgsql-16/bin/pgbench -c 8 -n -P 60 -T 60 -h localhost -p 54322 -U pgbenc  0.00s user 0.00s system 0% cpu 4:40.38 total



% pgagroal-cli status details
-> DEBUG management.c:1852 pgagroal-cli version 20000
Status:              Running
Active connections:  10
Total connections:   10
Max connections:     10
---------------------
Server:              rachel
Host:                127.0.0.1
Port:                5432
State:               Primary
---------------------
Database:            pgbench
Username:            pgbench
Active connections:  10
Max connections:     10
Initial connections: 5
Min connections:     5
---------------------
Connection    1:     Replica         2024-10-18 08:11:52 267875 20     pgbench pgbench pgbench
Connection    2:     Replica         2024-10-18 08:11:52 267876 21     pgbench pgbench pgbench
Connection    3:     Replica         2024-10-18 08:11:52 267877 22     pgbench pgbench pgbench
Connection    4:     Replica         2024-10-18 08:11:52 267878 23     pgbench pgbench pgbench
Connection    5:     Replica         2024-10-18 08:11:52 267879 24     pgbench pgbench pgbench
Connection    6:     Replica         2024-10-18 08:11:52 267880 25     pgbench pgbench pgbench
Connection    7:     Replica         2024-10-18 08:11:52 267882 25     pgbench pgbench pgbench
Connection    8:     Replica         2024-10-18 08:11:52 267884 25     pgbench pgbench pgbench
Connection    9:     Replica         2024-10-18 08:14:09 267896 25     pgbench pgbench pgbench
Connection   10:     Replica         2024-10-18 08:14:09 267897 26     pgbench pgbench pgbench


# amd if I force shutdown of pgagroal

-> INFO  main.c:1150 pgagroal: shutdown

% ps -auxw | grep pgagroal
luca      267875  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267876  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267877  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267878  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267879  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267880  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267882  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267884  0.2  0.2  13712  2772 pts/0    S    08:11   0:01 pgagroal: pgbench/pgbench
luca      267896  0.1  0.2  13712  2644 pts/0    S    08:14   0:00 pgagroal: pgbench/pgbench
luca      267897  0.1  0.2  13712  2644 pts/0    S    08:14   0:00 pgagroal: pgbench/pgbench

fluca1978 avatar Oct 18 '24 07:10 fluca1978

Thanks for the report, Luca. I believe this was fixed. Can you please confirm?

Uhm, I'm getting the very same behavior on 4df7ed6 (please note that the status of the connections is reported as replica, but this has been fixed, I think there is the need for a rebase).

Rebased.

Are you running pgbench commands concurrently, or do you run them one after another?

If you run them one after another, do the connections hang before pgbench runs, or after it completes?

Do you have any specific configuration (perhaps non-default settings) that might hint at why this behavior is happening?

I noticed the maximum number of connections is set to 10. Could this issue be related to #471? Meaning, are there enough resources available for the connections?

Edit: Have you tried running with the epoll backend? Does this behavior persist with epoll, or is it specific to io_uring?

hac-v avatar Oct 18 '24 10:10 hac-v

Are you running pgbench commands concurrently, or do you run them one after another?

No, I run a first pgbench that completes normally, than I start another one that hangs. If then I kill pgagroal, processes remain active.

If you run them one after another, do the connections hang before pgbench runs, or after it completes?

As I start the second run, it seems to hang, no progress is reported by pgbench, so it seems it is stuck.

Do you have any specific configuration (perhaps non-default settings) that might hint at why this behavior is happening?

No, I tried also changing the host setting to localhost in the case it was due to my dual home setup, but it does not change its behavior.

I noticed the maximum number of connections is set to 10. Could this issue be related to #471? Meaning, are there enough resources available for the connections?

Uhm, good guess, I can try to increase the max_connections and repeat the test, however, in #471 there is no problem as processes lying around after pgagroal has been killed.

Edit: Have you tried running with the epoll backend? Does this behavior persist with epoll, or is it specific to io_uring?

I did, and it worked fine, but I need to repeat the test to ensure there are no misconfiguration.

fluca1978 avatar Oct 20 '24 08:10 fluca1978

A little update on my tests. These are run after a machine reboot.

Using io_uring as event engine, max_connections = 20, while max connections on pgbench database is set to 10. I start pgagroal, then run pgbench -c 8, and it worked. Then pgench -c 8 again and it worked. The pgbench -c 10 (as the max connections for the database) and it hangs. Killed pgbench test, then run again pgbench -c 8 without stopping pgagroal and it hangs.

Meanwhile, on another terminal:

% pgagroal-cli status details
-> DEBUG management.c:1848 pgagroal-cli version 20000
Status:              Running
Active connections:  10
Total connections:   10
Max connections:     20
---------------------
Server:              rachel
Host:                127.0.0.1
Port:                5432
State:               Primary
---------------------
Database:            pgbench
Username:            pgbench
Active connections:  10
Max connections:     10
Initial connections: 5
Min connections:     5
---------------------
Connection    1:     Active          2024-10-21 08:37:41 3222   14     pgbench pgbench pgbench
Connection    2:     Active          2024-10-21 08:37:41 3220   15     pgbench pgbench pgbench
Connection    3:     Active          2024-10-21 08:37:41 3221   16     pgbench pgbench pgbench
Connection    4:     Active          2024-10-21 08:37:41 3223   17     pgbench pgbench pgbench
Connection    5:     Active          2024-10-21 08:37:41 3224   18     pgbench pgbench pgbench
Connection    6:     Active          2024-10-21 08:37:41 3225   19     pgbench pgbench pgbench
Connection    7:     Active          2024-10-21 08:37:41 3226   13     pgbench pgbench pgbench
Connection    8:     Active          2024-10-21 08:37:41 3227   20     pgbench pgbench pgbench
Connection    9:     Active          2024-10-21 08:39:02 3260   22     pgbench pgbench pgbench
Connection   10:     Active          2024-10-21 08:39:01 3258   22     pgbench pgbench pgbench
Connection   11:     Not initialized                                     
Connection   12:     Not initialized                                     
Connection   13:     Not initialized                                     
Connection   14:     Not initialized                                     
Connection   15:     Not initialized                                     
Connection   16:     Not initialized                                     
Connection   17:     Not initialized                                     
Connection   18:     Not initialized                                     
Connection   19:     Not initialized                                     
Connection   20:     Not initialized               

% psql -U postgres -h localhost -c 'select count(*), wait_event, application_name from pg_stat_activity where datname = $$pgbench$$ group by 2,3' pgbench
 count | wait_event | application_name 
-------+------------+------------------
     5 | ClientRead | pgbench

At this point, killing pgagroal leaves processes behind, while performing pgagroal-cli shutdown does not), so SIGINT has a wrong behavior:

% ps -auxw | grep pgagroal
luca        3220  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3221  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3222  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3223  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3224  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3225  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3226  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3227  0.2  0.3  13768  2900 pts/0    S    08:37   0:01 pgagroal: pgbench/pgbench
luca        3258  0.0  0.2  13768  2772 pts/0    S    08:39   0:00 pgagroal: pgbench/pgbench
luca        3260  0.0  0.3  13768  2900 pts/0    S    08:39   0:00 pgagroal: pgbench/pgbench

Switched to epoll and retsarted pgagroal. The launched pgbench -c 8 and it hangs. Meanwhile:

% psql -U postgres -h localhost -c 'select count(*), wait_event, application_name from pg_stat_activity where datname = $$pgbench$$ group by 2,3' pgbench
 count | wait_event | application_name 
-------+------------+------------------
     5 | ClientRead | pgbench

so no connections attempted at all, neither the min for the dtaabase (that is set to five). Here, killing pgagroal does not leave any process behind.

Then I switched to master, and restarted pgagroal. Run pgbench -c 8 and it worked. Run pgbench -c 8 again, and it worked. Run pgbench -c 10 and it worked, so this is where io_uring hangs (limit of the database, not max_connections). Run again pgbench -c 10 and it worked. Run pgbench -c 20 (hit max_connections limit) and it hangs, so this is related to #471.

Another test: if I run pgbench -c 10 immediatly after a pgagroal start, all works. So it seems that io_ruing is not releasing connections or some other resource, therefore preventing another run. In fact, if I then start again pgbench, no matter how many clients I ask for, the application hangs.

Last but not least, after the machine reboot, I started pgagroal with io_uring, run pgbench and everything was frozen, as it was hanging. I then realezied that I had not persisted the sysctl kernel.io_uring_disabled=0. Not sure if this is related, but the appearing behavior was the same as when the max database limit is hit.

Below my configuration:

% cat /etc/pgagroal.conf
[rachel]
host = 127.0.0.1
port = 5432
primary = on



[pgagroal]
host = localhost
port = 54322
unix_socket_dir = /tmp
pidfile = foo.pid

management=6432

log_type  = console
log_level = debug
#log_level = trace
log_path  = /var/log/pgagroal/pgagroal.log
log_path  = /var/log/pgagroal/pgagroal-%Y-%m-%d-%H-%M-%S.log   
log_mode  = create

log_connections    = on
log_disconnections = on
log_rotation_size = "3Mb"# in megabytes
log_rotation_age = 1m# one minute
log_line_prefix  = "->" #  "PGAGROAL #%Y-%m-%d-%H:%M:%S" # the prefix of the log

max_connections = 20

pipeline = 'performance'
# solo per la pipeline transaction
allow_unknown_users = false
blocking_timeout = 10000

metrics = 8000
metrics_cache_max_age = 10s
metrics_cache_max_size = 50Kb

#idle_timeout = 0

update_process_title = minimal

#master_key_file_location = /home/luca/tmp


ev_backend = io_uring # io_uring, epoll, kqueue.

and the database connection limits

% cat /etc/pgagroal/pgagroal_databases.conf
# DATABASE USER MAX INITIAL MIN
 pgbench pgbench 10 5  5

fluca1978 avatar Oct 21 '24 07:10 fluca1978

Thanks. This was very helpful. I will take a better look later tonight.

hac-v avatar Oct 21 '24 12:10 hac-v