Kamil Braun comments

Results 552 comments of


                                            Kamil Braun

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

According to node-3 the abort_requested originated at node-7, and there we can see: ``` Apr 13 05:20:47.635055 longevity-parallel-topology-schema--db-node-4bdf0ea1-7 scylla[7304]: [shard 2:strm] repair - repair[7c1d569e-55dc-44df-a701-44028be68f69]: 2 out of 576 ranges failed,...

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

Heh, isn't this just https://github.com/scylladb/scylladb/issues/15598, except we're getting `abort_requested_exception` instead of `no_such_column_family` for some reason? cc @Deexie @denesb

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

@Deexie why does the regression test in 2ea5d9b62327eb20727205d534f8112558518c45 disable repair-based ops? repair-based ops are the default

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

I suspect the exception might be flying out of this `abort_on_expiry` here: ```cpp future table_sync_and_check(replica::database& db, service::migration_manager& mm, const table_id& uuid) { if (mm.use_raft()) { abort_on_expiry aoe(lowres_clock::now() + std::chrono::seconds{10}); auto&...

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

I wonder whether this DROP INDEX actually finishes, or perhaps it's waiting for something. And if it finishes -- how long does it take. It seems plausible that we reach...

`nodetool removenode` may fail if we concurrently try to do `DROP INDEX...` resulting in leaving a dead DB node as part of the DB cluster

Closing as duplicate of https://github.com/scylladb/scylladb/issues/18452.

Decommission failed with error: remote replied with status code 500 Internal Server Error and coredumps due to possible reason that new node marked UN earler during bootstrap and replace with MV/SI

Guys why are you inlining those coredump details in your posts? ``` Found module libgnutls.so.30 with build-id: ca46bc2e9c1c40fe0c42ec412eb02187782b6cf2 Metadata for module libgnutls.so.30 owned by FDO found: { "type" : "rpm",...

Decommission failed with error: remote replied with status code 500 Internal Server Error and coredumps due to possible reason that new node marked UN earler during bootstrap and replace with MV/SI

@gleb-cloudius Is the diff in mutate_MV needed at all? ``` - static constexpr size_t max_concurrent_updates = 128; + static constexpr size_t max_concurrent_updates = 2; + co_await seastar::sleep(std::chrono::milliseconds{100}); ``` I understand...

Decommission failed with error: remote replied with status code 500 Internal Server Error and coredumps due to possible reason that new node marked UN earler during bootstrap and replace with MV/SI

Given the evidence (https://github.com/scylladb/scylladb/issues/17786#issuecomment-2115281296) it's most likely a regression introduced in https://github.com/scylladb/scylladb/pull/17149, so reassigning to @bhalevy.

Decommission failed with error: remote replied with status code 500 Internal Server Error and coredumps due to possible reason that new node marked UN earler during bootstrap and replace with MV/SI

> This needs to be coordinated with https://github.com/scylladb/scylladb/issues/18709. The fix may be the same. But this one points to a regression inside effective_replication_map construction code which should construct them atomically...