Israel Fruchter
Israel Fruchter
> @fruch , that's different. This time it's RF=1 case and, respectively, some other error. > Though the root-cause is the same -- node being decommissioned drops in the middle...
> Yes, this particular test case (RF=1) also needs retry policy By default there retry, but it does it with any wait time between each try In the latest driver...
coredump information (wasn't enough room for it in the description) ``` 2024-02-19 23:22:41.652 : (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=bc09dcee-d52c-4079-af57-7d6cc02bfccf during_nemesis=CreateIndex,RemoveNodeThenAddNode node=Node longevity-parallel-topology-schema--db-node-906aea29-2 [18.201.166.19 | 10.4.10.92] (seed: True) corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.112.93d7d7a7395d4476b644e35b602e4f23.6134.1708383810000000/core.scylla.112.93d7d7a7395d4476b644e35b602e4f23.6134.1708383810000000.gz backtrace= PID: 6134...
Not sure I can validate it, but I'm guess it's one more thing related to: https://github.com/scylladb/scylladb/issues/15805
> Do we have the full coredump for analysis? Yes download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.112.93d7d7a7395d4476b644e35b602e4f23.6134.1708383810000000/core.scylla.112.93d7d7a7395d4476b644e35b602e4f23.6134.1708383810000000.gz .
> @fruch - any idea what changed? Anything new in the test? (could it be due to my change with SCT - [scylladb/scylla-cluster-tests#7191](https://github.com/scylladb/scylla-cluster-tests/issues/7191) ?!?!) Asking is I'm unsure why we...
> @fruch - the key here is the constant load, for so many hours, that we may not have had before. again, it was removing a node, with `nodetool removenode`...
seems this is still happening, i.e. reproducible to some degree from the thread, sound like it's a relatively small fix, is there any known risk with it ? or what...
and happen on one more case this weeked: ## Packages Scylla version: `5.5.0~dev-20240329.885cb2af07b8` with build-id `4d1fc3fe8868b3d00a42f2b0b51f9953e9fa7346` Kernel Version: `5.15.0-1056-aws` ## Installation details Cluster size: 12 nodes (i3en.2xlarge) Scylla Nodes used...