neon icon indicating copy to clipboard operation
neon copied to clipboard

failed to run pgbench in docker setup

Open LizardWizzard opened this issue 4 years ago • 2 comments

I'm running pgbench -s5 -i in docker setup as it is in our console repo.

pgbench output:

NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
500000 of 500000 tuples (100%) done (elapsed 0.28 s, remaining 0.00 s)
vacuuming...
pgbench: fatal: query failed: ERROR:  could not read block 0 in rel 1663/16385/16399.2 from page server at lsn 0/08D8C7A8
CONTEXT:  while scanning block 0 of relation "public.pgbench_branches"
pgbench: query was: vacuum analyze pgbench_branches

In pageserver logs:

INFO Created database 1663/16385, copied 1014 blocks in 243 rels at 0/169F110, location: pageserver::restore_local_repo, pageserver/src/restore_local_repo.rs:646
INFO switched segment 0/1FFEB00 to 0/2002660, pageserver/src/walreceiver.rs:257
INFO switched segment 0/2FFFDF8 to 0/301D870, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/3FFF090 to 0/4004C40, pageserver/src/walreceiver.rs:257
INFO switched segment 0/4FFE860 to 0/5005EF8, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/5FFFAF8 to 0/6003658, pageserver/src/walreceiver.rs:257
INFO switched segment 0/6FFED28 to 0/7000D88, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/7FFFF80 to 0/8001FE0, pageserver/src/walreceiver.rs:257
WARN drop_segment called on non-existent relish 1663/16385/16389 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16389_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16389_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D8C7A8 to arrive, pageserver/src/page_service.rs:377

Timeouts are probably fine, because I had runs with timeouts but without this warnings

@lubennikovaav is it #502 or something new?

LizardWizzard avatar Sep 13 '21 19:09 LizardWizzard

Warning was fixed some time ago, wait lsn timeout error should be fixed in #630

LizardWizzard avatar Oct 01 '21 14:10 LizardWizzard

@LizardWizzard let's check that again since backpressure was merged/fixed? #803

kelvich avatar Dec 22 '21 09:12 kelvich

@LizardWizzard could this be closed ? or do we need it for anything else ?

shanyp avatar Feb 23 '23 10:02 shanyp

Lets close this in favor of https://github.com/neondatabase/neon/issues/1587.

Current issue was discovered when I was running whole setup in docker on mac os, so it was quite slow. I no longer have this setup at hand, so I think it makes sense to add additional test for backpressure with CPU limits applied for pageserver process

LizardWizzard avatar Feb 27 '23 10:02 LizardWizzard