scylladb icon indicating copy to clipboard operation
scylladb copied to clipboard

generic_server: Don't mess with db::config

Open xemul opened this issue 1 year ago • 11 comments

The db::config is top-level configuration of scylla, we generally try to avoid using it even in scylla components: each uses its own config initialized by the service creator out of the db::config itself. The generic_server is not an exception, all the more so, it already has its own config.

xemul avatar Apr 11 '25 12:04 xemul

:red_circle: CI State: FAILURE

:white_check_mark: - Framework test :x: - Build

Build Details:

  • Duration: 1 hr 8 min
  • Builder: spider4.cloudius-systems.com

scylladb-promoter avatar Apr 11 '25 13:04 scylladb-promoter

I don't understand why

[2025-04-11T13:33:42.244Z] ERROR: [ERROR]: [ERROR]: Build x86_64 failed with error: script returned exit code 137

no compilation errors found in logs. Will re-kick

xemul avatar Apr 11 '25 14:04 xemul

:red_circle: CI State: FAILURE

:white_check_mark: - Framework test :white_check_mark: - Build :white_check_mark: - Unit Tests Custom The following new/updated tests ran 100 times for each mode: :small_blue_diamond: boost/generic_server_test :white_check_mark: - dtest with tablets :white_check_mark: - dtest with consistent topology changes :white_check_mark: - dtest with gossip topology changes :x: - Unit Tests

Failed Tests (8/43818):

Build Details:

  • Duration: 11 hr
  • Builder: spider9.cloudius-systems.com

scylladb-promoter avatar Apr 12 '25 01:04 scylladb-promoter

CI failure:

The servers is asked to stop at 53:02

01:53:02.481 DEBUG>  ManagerClient stopping gracefully 5421
01:53:02.481 DEBUG>  RESTClient fetching PUT http://api/cluster/server/5421/stop_gracefully
01:53:02.481 DEBUG>  ManagerClient stopping gracefully 5422
01:53:02.481 DEBUG>  RESTClient fetching PUT http://api/cluster/server/5422/stop_gracefully

one server starts stopping instantly

INFO  2025-04-12 01:53:02,585 [shard 0:main] compaction_manager - Asked to stop
INFO  2025-04-12 01:53:02,585 [shard 0:main] view_update_generator - Terminating background fiber
INFO  2025-04-12 01:53:02,585 [shard 0:strm] view_update_generator - leaving 0 unstaged sstables unprocessed
WARN  2025-04-12 01:53:02,585 [shard 0:strm] gossip - failure_detector_loop: Got error in the loop, live_nodes={cd1dc062-56bd-4b29-8a55-77339270c4f2}: seastar::sleep_aborted (Sleep is aborted)
INFO  2025-04-12 01:53:02,585 [shard 0:strm] gossip - failure_detector_loop: Finished main loop
INFO  2025-04-12 01:53:02,585 [shard 0:main] init - Signal received; shutting down

but another one waits for ~40 seconds

INFO  2025-04-12 01:53:02,586 [shard 0: gms] gossip - Got shutdown message from 676e950a-2c42-4d37-b0e5-5731598f826d, received_generation=1744411973, local_generation=1744411973
INFO  2025-04-12 01:53:02,586 [shard 0: gms] gossip - InetAddress 676e950a-2c42-4d37-b0e5-5731598f826d is now DOWN, status = shutdown
INFO  2025-04-12 01:53:06,412 [shard 0:main] raft_group_registry - marking Raft server 676e950a-2c42-4d37-b0e5-5731598f826d as dead for raft groups
INFO  2025-04-12 01:53:39,734 [shard 0:main] compaction_manager - Asked to stop
INFO  2025-04-12 01:53:39,734 [shard 0:main] view_update_generator - Terminating background fiber
INFO  2025-04-12 01:53:39,734 [shard 0:strm] view_update_generator - leaving 0 unstaged sstables unprocessed
WARN  2025-04-12 01:53:39,734 [shard 0:strm] gossip - failure_detector_loop: Got error in the loop, live_nodes={}: seastar::sleep_aborted (Sleep is aborted)
INFO  2025-04-12 01:53:39,734 [shard 0:strm] gossip - failure_detector_loop: Finished main loop
INFO  2025-04-12 01:53:39,734 [shard 0:main] init - Signal received; shutting down

xemul avatar Apr 14 '25 07:04 xemul

:red_circle: CI State: FAILURE

:white_check_mark: - Framework test :white_check_mark: - Build :white_check_mark: - Unit Tests Custom The following new/updated tests ran 100 times for each mode: :small_blue_diamond: boost/generic_server_test :white_check_mark: - dtest with consistent topology changes :white_check_mark: - dtest with tablets :white_check_mark: - dtest with gossip topology changes :x: - Unit Tests

Failed Tests (2/43838):

Build Details:

  • Duration: 7 hr 26 min
  • Builder: spider9.cloudius-systems.com

scylladb-promoter avatar Apr 14 '25 14:04 scylladb-promoter

The very same symptoms as in #23731

xemul avatar Apr 15 '25 06:04 xemul

:red_circle: CI State: UNSTABLE

:white_check_mark: - Framework test :white_check_mark: - Build :white_check_mark: - Unit Tests Custom The following new/updated tests ran 100 times for each mode: :small_blue_diamond: boost/generic_server_test :white_check_mark: - dtest with tablets :white_check_mark: - dtest with gossip topology changes :x: - dtest with consistent topology changes :white_check_mark: - Unit Tests

Build Details:

  • Duration: 6 hr 39 min
  • Builder: spider3.cloudius-systems.com

scylladb-promoter avatar Apr 15 '25 12:04 scylladb-promoter

#22396

xemul avatar Apr 15 '25 17:04 xemul

:red_circle: CI State: FAILURE

:white_check_mark: - Framework test :white_check_mark: - Build :white_check_mark: - Unit Tests Custom The following new/updated tests ran 100 times for each mode: :small_blue_diamond: boost/generic_server_test :white_check_mark: - dtest with gossip topology changes :white_check_mark: - dtest with tablets :white_check_mark: - dtest with consistent topology changes :x: - Unit Tests

Failed Tests (1/43776):

Build Details:

  • Duration: 6 hr 53 min
  • Builder: spider4.cloudius-systems.com

scylladb-promoter avatar Apr 16 '25 00:04 scylladb-promoter

https://github.com/scylladb/scylladb/issues/23762

xemul avatar Apr 16 '25 07:04 xemul

:green_circle: CI State: SUCCESS

:white_check_mark: - Framework test :white_check_mark: - Build :white_check_mark: - Unit Tests Custom The following new/updated tests ran 100 times for each mode: :small_blue_diamond: boost/generic_server_test :white_check_mark: - dtest with tablets :white_check_mark: - dtest with gossip topology changes :white_check_mark: - dtest with consistent topology changes :white_check_mark: - Unit Tests

Build Details:

  • Duration: 6 hr 50 min
  • Builder: spider4.cloudius-systems.com

scylladb-promoter avatar Apr 16 '25 13:04 scylladb-promoter