Provision test fails due error in delete_by_range_using_timestamp nemesis

[soyacz]

Provision test for azure picsk delete_by_range_using_timestamp nemesis. Because it uses scylla-bench it is not skipped and fails with error:

13:55:21  2024-05-20 11:44:00.038: (DisruptionEvent Severity.ERROR) period_type=end event_id=f7575e80-f082-4305-ad64-aa3ed1a11dce duration=2s: nemesis_name=DeleteByRowsRange target_node=Node pr-provision-test-pr-7445-db-node-abf64be1-eastus-3 [172.191.133.206 | 10.0.0.7] errors=Unable to find partitions to delete
13:55:21  Traceback (most recent call last):
13:55:21  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5117, in wrapper
13:55:21  result = method(*args[1:], **kwargs)
13:55:21  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2431, in disrupt_delete_by_rows_range
13:55:21  partitions_for_exclude = self.delete_by_range_using_timestamp(ks_cf)
13:55:21  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2330, in delete_by_range_using_timestamp
13:55:21  raise PartitionNotFound(message)
13:55:21  sdcm.exceptions.PartitionNotFound: Unable to find partitions to delete

logs details:

+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                    Log links for testrun with test id abf64be1-5eb4-42de-ab3c-63a2047838c8                                                                                                    |
+-----------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Date            | Log type    | Link                                                                                                                                                                                                                                          |
+-----------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 20240520_114652 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_114652/grafana-screenshot-overview-20240520_114652-pr-provision-test-pr-7445-monitor-node-abf64be1-eastus-1.png                                  |
| 20240520_114652 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_114652/grafana-screenshot-pr-7445-scylla-per-server-metrics-nemesis-20240520_114819-pr-provision-test-pr-7445-monitor-node-abf64be1-eastus-1.png |
| 20240520_115537 | db-cluster  | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_115537/db-cluster-abf64be1.tar.gz                                                                                                                |
| 20240520_115537 | loader-set  | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_115537/loader-set-abf64be1.tar.gz                                                                                                                |
| 20240520_115537 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_115537/monitor-set-abf64be1.tar.gz                                                                                                               |
| 20240520_115537 | sct         | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_115537/sct-abf64be1.log.tar.gz                                                                                                                   |
| 20240520_115537 | event       | https://cloudius-jenkins-test.s3.amazonaws.com/abf64be1-5eb4-42de-ab3c-63a2047838c8/20240520_115537/sct-runner-events-abf64be1.tar.gz                                                                                                         |
+-----------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[fruch]

this is the validation done before using this nemeis:

    def verify_initial_inputs_for_delete_nemesis(self):
        test_keyspaces = self.cluster.get_test_keyspaces()

        if 'scylla_bench' not in test_keyspaces:
            raise UnsupportedNemesis("This nemesis can run on scylla_bench test only")

        if not (self.tester.partitions_attrs and self.tester.partitions_attrs.max_partitions_in_test_table):
            raise UnsupportedNemesis(
                'This nemesis expects "max_partitions_in_test_table" sub-parameter of data_validation to be set')

in provision test we use s-b, and that max_partitions_in_test_table seems to existing by default

this can need to change existence of a specific scylla-bench command, I guess it has hidden assumption it won't be running during prepare, or only when there is enough data

[fruch]

Also it has nothing todo with Azure, it can happen on any backend

[dimakr]

Another occurrence of this on AWS backend in CI - https://jenkins.scylladb.com/job/sct-github-PRs-scan/job/scylla-cluster-tests/job/PR-10585/3/

[dimakr]

One more occurrence of the issue: https://jenkins.scylladb.com/job/sct-github-PRs-scan/job/scylla-cluster-tests/job/PR-11310/7/consoleFull#-1083288395fcc21424-66d2-4bd8-8e0d-9746405e5b16