Runs for multiDC cluster from local development machine don't work

[mikliapko]

Issue description

• [ ] This issue is a regression.
• [x] It is unknown if this issue is a regression.

Attempt to run the test with multiDC cluster from local development machine using the config for local run from README:

hydra run-test mgmt_cli_test.MgmtCliTest.test_manager_sanity \
      --backend aws \
      --config test-cases/manager/manager-regression-multiDC-set-distro.yaml \
      --config configurations/network_config/test_communication_public.yaml

results in failure to start scylla-server:

...
Apr 29 11:14:46 manager-regression-mikita-db-node-132c0ce7-1 scylla[7251]:  [shard 0:main] init - seeds={10.12.3.46}, listen_address=10.12.3.46, broadcast_address=100.27.33.165
Apr 29 11:14:46 manager-regression-mikita-db-node-132c0ce7-1 scylla[7251]:  [shard 0:main] init - Use broadcast_address instead of listen_address for seeds list
...
Apr 29 11:14:46 manager-regression-mikita-db-node-132c0ce7-1 scylla[7251]:  [shard 0:main] init - Startup failed: std::runtime_error (Use broadcast_address for seeds list)

Expectations

• Provide the proper config / fix existing for local runs of multiDC cluster tests;
• Document the multiDC cluster local run procedure.

Impact

It blocks the execution of multiDC cluster tests from local development machine

How frequently does it reproduce?

100%

Installation details

SCT Version: master/e80bb05d5dba75779350f86704d8caedf1df0609 Scylla version (or git commit hash): 2024.1

Logs

• test_id: 132c0ce7-df5e-42c9-84a9-9366425ba9f1
• log archive: 20240429-105959-390832.zip

[fruch]

seem like this case is getting into the wrong snitch: Ec2MultiRegionSnitch

hence the snitch is picking the public address, which doesn't match the configuration.

@juliayakovlev this logic is wrong, the type is based on test_configuration, but it should be based on broadcast_address

    @property
    def _default_endpoint_snitch(self) -> Literal[
            'org.apache.cassandra.locator.Ec2MultiRegionSnitch',
            'org.apache.cassandra.locator.GossipingPropertyFileSnitch',
            'org.apache.cassandra.locator.Ec2Snitch']:
        if self._cluster_backend == 'aws':
            if ssh_connection_ip_type(self.params) == 'public':
                return 'org.apache.cassandra.locator.Ec2MultiRegionSnitch'
            else:
                return 'org.apache.cassandra.locator.Ec2Snitch'
        return 'org.apache.cassandra.locator.GossipingPropertyFileSnitch'

[fruch]

@mikliapko

I think adding this configuration might work for you locally

export SCT_ENDPOINT_SNITCH='Ec2Snitch'

[fruch]

FYI:

the log like show the snitch is picking the address:

< t:2024-04-29 11:14:45,731 f:db_log_reader.py l:123  c:sdcm.db_log_reader   p:DEBUG > 2024-04-29T11:14:45.667+00:00 manager-regression-mikita-db-node-132c0ce7-1     !INFO | scylla[7251]:  [shard 0:main] snitch_logger - Ec2MultiRegionSnitch using publicIP as identifier: 100.27.33.165

[juliayakovlev]

seem like this case is getting into the wrong snitch: Ec2MultiRegionSnitch

hence the snitch is picking the public address, which doesn't match the configuration.

@juliayakovlev this logic is wrong, the type is based on test_configuration, but it should be based on broadcast_address

I am wondering if it was worked ever

    @property
    def _default_endpoint_snitch(self) -> Literal[
            'org.apache.cassandra.locator.Ec2MultiRegionSnitch',
            'org.apache.cassandra.locator.GossipingPropertyFileSnitch',
            'org.apache.cassandra.locator.Ec2Snitch']:
        if self._cluster_backend == 'aws':
            if ssh_connection_ip_type(self.params) == 'public':
                return 'org.apache.cassandra.locator.Ec2MultiRegionSnitch'
            else:
                return 'org.apache.cassandra.locator.Ec2Snitch'
        return 'org.apache.cassandra.locator.GossipingPropertyFileSnitch'

[mikliapko]

@mikliapko

I think adding this configuration might work for you locally

export SCT_ENDPOINT_SNITCH='Ec2Snitch'

It actually helped me to start tests from the local machine, thanks