scylla-cluster-tests icon indicating copy to clipboard operation
scylla-cluster-tests copied to clipboard

Published 20 hours ago • verified publisher icon scylladb

[AWS] SCT calls `network_interfaces` node attr `4199` times for a cluster of 60 nodes making bootstrap be seriously delayed

Open vponomaryov opened this issue 9 months ago • 13 comments

Packages

Issue description

This issue is proper bug filing of the problem described here: https://github.com/scylladb/scylla-enterprise/issues/5250 SCT branch: branch-2025.1 (probably master branch is also affected)

Running a 5dc/60nodes test case the bootstrap of all the nodes took about 18hours but was expected to take about 4hours. And it is caused by the redundancy of getting network interfaces for each of the nodes.

The main problem is that starting with the 25th DB node SCT spends 23+ seconds for getting network interfaces info per node. And after each new DB addition to a DB cluster the process gets repeated.

SCT runner logs:

$ grep -R -c " ethernets: " sct-a45b2013.log
4199

Or 70 times per node where the last one is processed just 69 times.

Code part that has that substring:

    @property
    def network_configuration(self):
        # Output example:
        #   0a:7b:18:de:f9:71: eth0
        #   0a:7b:18:de:f9:72: eth1
        ip_link_cmd = """ip -o link | awk '$2 != "lo:" {gsub(/:/,"",$2);print $17": " $2}'"""
        network_config = self.remoter.run(ip_link_cmd).stdout.strip()
        network_devices = yaml.safe_load(network_config)
        self.log.debug("Node %s ethernets: %s", self.name, network_devices)
        return network_devices

It is located here: https://github.com/scylladb/scylla-cluster-tests/blob/d328c1d77d576096dc0281759ed269efccb1d61f/sdcm/cluster_aws.py#L481

On the DB logs side it looks like the following:

./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-1/system.log:Feb 19 12:43:32.920591 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-1 scylla[5519]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-2/system.log:Feb 19 12:44:50.820905 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-2 scylla[5529]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-3/system.log:Feb 19 12:45:43.091424 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-3 scylla[5548]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-4/system.log:Feb 19 12:46:36.698993 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-4 scylla[5551]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-5/system.log:Feb 19 12:47:33.233803 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-5 scylla[5568]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-6/system.log:Feb 19 12:48:27.323591 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-6 scylla[5575]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-7/system.log:Feb 19 12:49:22.266953 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-7 scylla[5590]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-8/system.log:Feb 19 12:50:19.683177 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-8 scylla[5606]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-9/system.log:Feb 19 12:51:18.756759 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-9 scylla[5622]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-10/system.log:Feb 19 12:52:17.609568 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-10 scylla[5626]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-11/system.log:Feb 19 12:53:20.466627 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-11 scylla[5652]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-12/system.log:Feb 19 12:54:30.033677 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-12 scylla[5661]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
<new region>
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-13/system.log:Feb 19 13:03:00.477954 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-13 scylla[5675]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-14/system.log:Feb 19 13:04:09.952189 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-14 scylla[5790]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-15/system.log:Feb 19 13:05:18.516489 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-15 scylla[5799]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-16/system.log:Feb 19 13:06:28.753776 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-16 scylla[5805]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-17/system.log:Feb 19 13:07:40.519627 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-17 scylla[5821]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-18/system.log:Feb 19 13:08:49.786992 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-18 scylla[5828]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-19/system.log:Feb 19 13:09:59.814050 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-19 scylla[5856]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-20/system.log:Feb 19 13:11:12.669817 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-20 scylla[5865]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-21/system.log:Feb 19 13:12:29.179821 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-21 scylla[5881]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-22/system.log:Feb 19 13:13:44.897420 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-22 scylla[5905]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-23/system.log:Feb 19 13:15:01.555419 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-23 scylla[5921]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-24/system.log:Feb 19 13:16:28.598370 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-24 scylla[5941]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
<new region>
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-25/system.log:Feb 19 13:44:19.864635 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-25 scylla[5961]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-26/system.log:Feb 19 14:08:11.377714 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-26 scylla[6581]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-27/system.log:Feb 19 14:31:44.383385 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-27 scylla[6883]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-28/system.log:Feb 19 14:55:19.541827 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-28 scylla[8242]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-29/system.log:Feb 19 15:18:57.561750 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-29 scylla[7462]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-30/system.log:Feb 19 15:42:35.713533 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-30 scylla[7756]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-31/system.log:Feb 19 16:06:21.248065 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-31 scylla[8104]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-32/system.log:Feb 19 16:30:12.452237 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-32 scylla[8332]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-33/system.log:Feb 19 16:54:05.699577 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-33 scylla[8657]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-34/system.log:Feb 19 17:17:58.370689 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-34 scylla[8931]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-35/system.log:Feb 19 17:41:57.142351 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-35 scylla[9216]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-36/system.log:Feb 19 18:06:08.837254 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-36 scylla[9527]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
<new region>
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-37/system.log:Feb 19 18:58:59.438914 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-37 scylla[9841]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-38/system.log:Feb 19 19:23:06.459261 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-38 scylla[10451]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-39/system.log:Feb 19 19:46:58.198339 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-39 scylla[10761]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-40/system.log:Feb 19 20:10:48.414966 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-40 scylla[11043]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-41/system.log:Feb 19 20:34:41.038574 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-41 scylla[11350]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-42/system.log:Feb 19 20:58:41.763179 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-42 scylla[11656]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-43/system.log:Feb 19 21:22:47.414416 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-43 scylla[11957]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-44/system.log:Feb 19 21:46:58.460026 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-44 scylla[12260]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-45/system.log:Feb 19 22:11:12.611175 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-45 scylla[12560]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-46/system.log:Feb 19 22:35:32.247608 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-46 scylla[12865]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-47/system.log:Feb 19 22:59:56.856165 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-47 scylla[13171]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-48/system.log:Feb 19 23:24:39.977870 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-48 scylla[13461]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
<new region>
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-49/system.log:Feb 20 02:11:50.126204 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-49 scylla[13939]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-50/system.log:Feb 20 02:20:27.174069 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-50 scylla[14423]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-51/system.log:Feb 20 02:16:10.264044 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-51 scylla[15629]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-52/system.log:Feb 20 02:44:01.125430 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-52 scylla[15692]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-53/system.log:Feb 20 03:11:35.666046 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-53 scylla[16246]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-54/system.log:Feb 20 03:39:16.453112 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-54 scylla[16362]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-55/system.log:Feb 20 04:07:14.011614 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-55 scylla[16806]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-56/system.log:Feb 20 04:35:11.156790 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-56 scylla[17035]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-57/system.log:Feb 20 05:03:24.295450 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-57 scylla[17449]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-58/system.log:Feb 20 05:31:45.579014 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-58 scylla[17816]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-59/system.log:Feb 20 06:00:17.054789 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-59 scylla[18518]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.
./long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-60/system.log:Feb 20 06:29:23.290501 long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-60 scylla[18435]:  [shard 0:main] init - Scylla version 2025.1.0~rc2-0.20250216.6ee17795783f initialization completed.

Impact

With increase of node count in a test cluster we get significant delays among points in time where SCT triggers a DB node startup.

How frequently does it reproduce?

1/1

Installation details

Cluster size: 60 nodes (i3en.large)

Scylla Nodes used in this run:

  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-9 (35.158.205.95 | 10.11.6.101) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-8 (18.197.151.189 | 10.11.5.113) (shards: -1)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-7 (3.78.187.98 | 10.11.5.200) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-60 (52.90.5.181 | 10.12.6.136) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-6 (18.196.96.121 | 10.11.5.212) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-59 (44.211.168.191 | 10.12.7.201) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-58 (54.84.165.202 | 10.12.4.24) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-57 (54.164.107.24 | 10.12.4.95) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-56 (54.86.31.247 | 10.12.4.112) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-55 (52.71.16.122 | 10.12.7.44) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-54 (44.211.250.189 | 10.12.7.39) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-53 (34.239.109.197 | 10.12.4.184) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-52 (52.70.135.167 | 10.12.6.148) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-51 (34.227.13.45 | 10.12.7.73) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-50 (52.70.25.135 | 10.12.5.181) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-5 (18.192.66.178 | 10.11.6.232) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-49 (54.161.226.78 | 10.12.5.213) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-48 (18.169.52.21 | 10.3.7.166) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-47 (13.40.81.6 | 10.3.7.117) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-46 (13.40.182.255 | 10.3.5.46) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-45 (13.40.211.97 | 10.3.7.106) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-44 (13.40.213.14 | 10.3.7.13) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-43 (3.10.59.27 | 10.3.7.118) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-42 (18.175.234.45 | 10.3.7.189) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-41 (18.168.206.179 | 10.3.4.150) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-40 (18.130.25.187 | 10.3.6.51) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-4 (54.93.97.94 | 10.11.5.0) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-39 (52.56.68.114 | 10.3.6.72) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-38 (18.130.217.29 | 10.3.4.98) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-37 (35.179.182.187 | 10.3.6.183) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-36 (34.255.119.141 | 10.4.5.45) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-35 (34.241.15.210 | 10.4.5.214) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-34 (54.195.51.251 | 10.4.4.156) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-33 (18.200.238.252 | 10.4.7.193) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-32 (34.251.50.145 | 10.4.5.48) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-31 (34.245.2.129 | 10.4.6.87) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-30 (54.170.127.200 | 10.4.6.90) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-3 (3.78.227.178 | 10.11.6.176) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-29 (3.255.159.117 | 10.4.4.121) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-28 (52.19.210.96 | 10.4.7.98) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-27 (34.250.140.46 | 10.4.6.228) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-26 (3.252.130.215 | 10.4.5.124) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-25 (34.247.84.92 | 10.4.7.179) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-24 (51.20.41.214 | 10.0.6.172) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-23 (13.60.214.165 | 10.0.4.76) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-22 (51.21.182.244 | 10.0.5.102) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-21 (13.60.157.41 | 10.0.7.119) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-20 (13.60.187.156 | 10.0.6.164) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-2 (18.184.29.206 | 10.11.4.174) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-19 (16.171.250.194 | 10.0.5.70) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-18 (13.60.92.182 | 10.0.7.151) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-17 (13.60.169.88 | 10.0.7.38) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-16 (51.21.167.165 | 10.0.6.81) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-15 (13.61.26.217 | 10.0.4.252) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-14 (13.60.248.215 | 10.0.7.220) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-13 (13.53.106.104 | 10.0.6.210) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-12 (3.79.33.163 | 10.11.5.67) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-11 (18.195.52.10 | 10.11.6.30) (shards: 2)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-10 (18.196.2.64 | 10.11.7.249) (shards: -1)
  • long-custom-d2-wrkld1-5dc-2025-1-db-node-a45b2013-1 (52.59.242.150 | 10.11.7.230) (shards: 2)

OS / Image: ami-0de818ccfeca21eed ami-00cc845d8d69ab68b ami-00dd94413096f919a ami-0d30c4beca10fc073 ami-05507142049c923a4 (aws: eu-central-1, eu-north-1, eu-west-1, eu-west-2, us-east-1)

Test: longevity-aws-custom-d2-workload1-5dcs-test Test id: a45b2013-902c-4445-af9e-7ba62da152ba Test name: scylla-2025.1/longevity/longevity-aws-custom-d2-workload1-5dcs-test Test method: longevity_test.LongevityTest.test_custom_time Test config file(s):

Show all stored logs command: $ hydra investigate show-logs a45b2013-902c-4445-af9e-7ba62da152ba

Jenkins job URL Argus

vponomaryov avatar Feb 27 '25 16:02 vponomaryov

@fruch this is blocker for the custom-d2/w1 scenario and any other that runs multi-dc multiple nodes.

The staging test run that didn't suffer from the slow bootstrap but still had lots of calls: scylla-staging/valerii/vp-longevity-aws-custom-d2-workload1-multidc-big#19 It's SCT parent commit history is following: https://github.com/scylladb/scylla-cluster-tests/commits/b4f6f58f44759e19d6733d0fd39a5d874b2bc01b/ (Dec 3, 2024) It has 23000 of the network_configuration calls, but these are fast enough in that case.

vponomaryov avatar Feb 27 '25 16:02 vponomaryov

I guess we can easily cache this property, it's not something that can change during a test, we never reconfigure networks in the middle of a test

fruch avatar Feb 27 '25 17:02 fruch

I guess we can easily cache this property, it's not something that can change during a test, we never reconfigure networks in the middle of a test

Yes, we can.

We call it 70 times per node in a 60 Db node cluster. So, something is really wrong here.

Upd: Looked at the code more, looks like we call that network interfaces code each time we do something with IP addresses where the _refresh_instance_state method gets called.

vponomaryov avatar Feb 27 '25 17:02 vponomaryov

I guess we can easily cache this property, it's not something that can change during a test, we never reconfigure networks in the middle of a test

Yes, we can.

We call it 70 times per node in a 60 Db node cluster. So, something is really wrong here.

Upd: Looked at the code more, looks like we call that network interfaces code each time we do something with IP addresses where the _refresh_instance_state method gets called.

So we definitely should cache it

fruch avatar Feb 28 '25 06:02 fruch

Due to public IP address changes after nemesis-initiated node restarts, caching mechanisms become ineffective, and reconfiguration is required. This is the reason why I did not cache it

juliayakovlev avatar Mar 02 '25 07:03 juliayakovlev

Due to public IP address changes after nemesis-initiated node restarts, caching mechanisms become ineffective, and reconfiguration is required. This is the reason why I did not cache it

but this part if for getting the mapping of private address to mac address, that isn't changing after restart

fruch avatar Mar 02 '25 07:03 fruch

Due to public IP address changes after nemesis-initiated node restarts, caching mechanisms become ineffective, and reconfiguration is required. This is the reason why I did not cache it

but this part if for getting the mapping of private address to mac address, that isn't changing after restart

Which part do you mean? _refresh_instance_state` ?

juliayakovlev avatar Mar 02 '25 08:03 juliayakovlev

the property network_configuration that @vponomaryov is pointing to in this issue.

that can be cached, isn't it ?

fruch avatar Mar 02 '25 08:03 fruch

the property network_configuration that @vponomaryov is pointing to in this issue.

that can be cached, isn't it ?

The public IP is refreshed there. If do not perform this, public IP will be wrong in the network interface

juliayakovlev avatar Mar 02 '25 08:03 juliayakovlev

Last Friday I ran a test using cached network interfaces. Test run: scylla-2025.1/longevity/longevity-aws-custom-d2-workload1-5dcs-test#3

Brief results, attention to the time diff between node-48 and node-49:

$ grep -R " initialization completed" . | grep system.log | sort

./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-1/system.log:Feb 28 10:05:51.290964 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-1 scylla[5511]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-2/system.log:Feb 28 10:06:11.541661 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-2 scylla[5513]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-3/system.log:Feb 28 10:06:27.123768 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-3 scylla[5538]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-4/system.log:Feb 28 10:06:46.448214 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-4 scylla[5519]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-5/system.log:Feb 28 10:07:06.719320 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-5 scylla[5515]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-6/system.log:Feb 28 10:07:26.855675 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-6 scylla[5526]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-7/system.log:Feb 28 10:07:46.637240 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-7 scylla[5527]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-8/system.log:Feb 28 10:08:07.351132 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-8 scylla[5529]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-9/system.log:Feb 28 10:08:28.772937 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-9 scylla[5528]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-10/system.log:Feb 28 10:08:56.555036 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-10 scylla[5546]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-11/system.log:Feb 28 10:09:22.073826 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-11 scylla[5534]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-12/system.log:Feb 28 10:09:52.700967 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-12 scylla[5539]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.

./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-13/system.log:Feb 28 10:17:41.891166 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-13 scylla[5551]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-14/system.log:Feb 28 10:18:14.055836 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-14 scylla[5635]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-15/system.log:Feb 28 10:18:45.683824 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-15 scylla[5637]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-16/system.log:Feb 28 10:19:16.778922 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-16 scylla[5641]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-17/system.log:Feb 28 10:19:48.588010 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-17 scylla[5655]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-18/system.log:Feb 28 10:20:20.266670 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-18 scylla[5654]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-19/system.log:Feb 28 10:20:54.303993 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-19 scylla[5665]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-20/system.log:Feb 28 10:21:31.231637 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-20 scylla[5671]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-21/system.log:Feb 28 10:22:09.051832 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-21 scylla[5675]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-22/system.log:Feb 28 10:22:47.301992 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-22 scylla[6694]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-23/system.log:Feb 28 10:23:28.880766 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-23 scylla[5692]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-24/system.log:Feb 28 10:24:19.963221 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-24 scylla[5695]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.

./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-25/system.log:Feb 28 10:50:50.320071 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-25 scylla[5707]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-26/system.log:Feb 28 10:52:13.182167 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-26 scylla[6007]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-27/system.log:Feb 28 10:53:13.228937 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-27 scylla[6047]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-28/system.log:Feb 28 10:54:17.468206 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-28 scylla[6024]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-29/system.log:Feb 28 10:55:21.912748 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-29 scylla[6041]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-30/system.log:Feb 28 10:56:26.581781 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-30 scylla[6049]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-31/system.log:Feb 28 10:57:39.385544 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-31 scylla[6062]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-32/system.log:Feb 28 10:58:58.060822 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-32 scylla[6068]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-33/system.log:Feb 28 11:00:19.191662 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-33 scylla[6085]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-34/system.log:Feb 28 11:01:41.790649 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-34 scylla[6100]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-35/system.log:Feb 28 11:03:06.276757 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-35 scylla[6127]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-36/system.log:Feb 28 11:04:46.942284 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-36 scylla[6129]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.

./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-37/system.log:Feb 28 11:34:51.218258 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-37 scylla[6151]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-38/system.log:Feb 28 11:36:25.660102 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-38 scylla[6497]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-39/system.log:Feb 28 11:37:35.994964 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-39 scylla[6501]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-40/system.log:Feb 28 11:38:48.486033 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-40 scylla[6530]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-41/system.log:Feb 28 11:40:08.528053 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-41 scylla[6525]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-42/system.log:Feb 28 11:41:30.057235 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-42 scylla[6558]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-43/system.log:Feb 28 11:42:58.011697 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-43 scylla[6562]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-44/system.log:Feb 28 11:44:33.372103 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-44 scylla[6577]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-45/system.log:Feb 28 11:46:09.233063 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-45 scylla[6595]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-46/system.log:Feb 28 11:47:48.642252 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-46 scylla[6613]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-47/system.log:Feb 28 11:49:37.448838 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-47 scylla[6636]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-48/system.log:Feb 28 11:51:48.092517 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-48 scylla[6649]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.

./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-49/system.log:Feb 28 14:16:30.586414 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-49 scylla[6653]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-50/system.log:Feb 28 14:20:29.900448 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-50 scylla[7346]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-51/system.log:Feb 28 14:24:32.091463 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-51 scylla[8014]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-52/system.log:Feb 28 14:29:31.224179 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-52 scylla[8383]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-53/system.log:Feb 28 14:34:22.465913 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-53 scylla[8440]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-54/system.log:Feb 28 14:39:21.004427 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-54 scylla[8477]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-55/system.log:Feb 28 14:44:25.514837 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-55 scylla[8560]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-56/system.log:Feb 28 14:49:48.694375 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-56 scylla[9394]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-57/system.log:Feb 28 14:55:18.697379 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-57 scylla[8675]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-58/system.log:Feb 28 15:00:53.763472 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-58 scylla[9564]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-59/system.log:Feb 28 15:06:36.607017 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-59 scylla[9638]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.
./long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-60/system.log:Feb 28 15:12:59.073616 long-custom-d2-wrkld1-5dc-vp-branc-db-node-0acb424e-60 scylla[8845]:  [shard 0:main] init - Scylla version 2025.1.0~rc3-0.20250223.aa5cb15166d3 initialization completed.

2025-02-28 15:38:03,668 f:tester.py       l:3020 c:LongevityTest        p:INFO  > TearDown is starting...

vponomaryov avatar Mar 03 '25 10:03 vponomaryov

seems like the mapping that was introduced in: https://github.com/scylladb/scylla-cluster-tests/pull/5869/

might contribute to some of the slowdown, since it calls get_all_ip_addresses() for each not every time it created

we might need to look into a way to update parts of this mapping

fruch avatar Mar 03 '25 10:03 fruch

seems like the mapping that was introduced in: #5869

might contribute to some of the slowdown, since it calls get_all_ip_addresses() for each not every time it created

we might need to look into a way to update parts of this mapping

It comes out to be the slow repair. Filed new bug: https://github.com/scylladb/scylla-enterprise/issues/5256

SCT does have inefficiency, but it was not the root cause of the main bootstrap slowliness in here: scylla-2025.1/longevity/longevity-aws-custom-d2-workload1-5dcs-test#3

vponomaryov avatar Mar 03 '25 18:03 vponomaryov

let's wait and see how it would be the fix (it's a one liner, I hope we'll get it fast enough)

not sure it would worth our while to refactor that part now.

fruch avatar Mar 03 '25 21:03 fruch