Kamil Braun comments

Results 554 comments of


                                            Kamil Braun

Scale test (adding 40 nodes) failed by timeouts(nodes added to long) and errors on teardown: seastar::rpc::remote_verb_error (Mutation of 17696794 bytes is too large for the maximum size of 16777216)

> > So @aleksbykov are you saying that test was passing for some time, but then there was a regression? > > yes, i pretty sure, that i run 1...

Scale test (adding 40 nodes) failed by timeouts(nodes added to long) and errors on teardown: seastar::rpc::remote_verb_error (Mutation of 17696794 bytes is too large for the maximum size of 16777216)

Could also be that before, snapshot transfer was retrying indefinitely and for some reason, would eventually succeed... But now we have a timeout on node bootstrap read barrier: ``` ('node36',...

Scale test (adding 40 nodes) failed by timeouts(nodes added to long) and errors on teardown: seastar::rpc::remote_verb_error (Mutation of 17696794 bytes is too large for the maximum size of 16777216)

> Isn't cdc_generations_v3 a raft table? Shouldn't it be using the schema commitlog? It is and it is

Scale test (adding 40 nodes) failed by timeouts(nodes added to long) and errors on teardown: seastar::rpc::remote_verb_error (Mutation of 17696794 bytes is too large for the maximum size of 16777216)

No. For example, if we created e.g. 3 generations in the span of a minute, then we need to transfer all 3. We have a mechanism to garbage-collect generations after...

Scale test (adding 40 nodes) failed by timeouts(nodes added to long) and errors on teardown: seastar::rpc::remote_verb_error (Mutation of 17696794 bytes is too large for the maximum size of 16777216)

Then let's reduce the 24h. Here's the explanation: ```cpp // If some node can still accept a write to a CDC generation, we cannot delete this generation. // A request...

Startup failed: std::runtime_error (A node with address x.x.x.x already exists)

So it looks like you have a joining node with this address, according to nodetool status output: ``` UJ x.x.x.x ? 256 ? null Rack1 ``` Are the logs of...

Startup failed: std::runtime_error (A node with address x.x.x.x already exists)

So it looks like you're automatically restarting the .6 node and it's continuously retrying to bootstrap? I see that the "already exists" failure is not the only one. Sometimes the...

Startup failed: std::runtime_error (A node with address x.x.x.x already exists)

Actually the timeout does not happen in repair, it happens inside a reader on the source node (.239), and then is propagated to repair code (cc @denesb ) ``` WARN...

Startup failed: std::runtime_error (A node with address x.x.x.x already exists)

There are lots of tombstones although in different tables ``` WARN 2024-05-06 22:14:30,389 [shard 3:stat] querier - Read 19 live rows and 1046 tombstones for sky_items_movement.items partition key "WISE_WITHER_BOOTS" {{4934172726220330794,...

Startup failed: std::runtime_error (A node with address x.x.x.x already exists)

There are indeed some large partitions there ``` INFO 2024-05-06 22:03:29,339 [shard 4:stre] compaction_manager - Starting off-strategy compaction for sky_auctions.auctions_highestbidder_idx_index compaction_group=0/1, 2 candidates were found INFO 2024-05-06 22:03:29,339 [shard 4:stre]...