CI Failure (DNS failure: communications error to 127.0.0.11#53) in `AlterTopicConfiguration.test_shadow_indexing_config`

[vbotbuildovich]

https://buildkite.com/redpanda/redpanda/builds/45930#018e251b-8649-4be6-96d3-6c47a0c923bb

Module: rptest.tests.alter_topic_configuration_test
Class: AlterTopicConfiguration
Method: test_shadow_indexing_config

test_id:    AlterTopicConfiguration.test_shadow_indexing_config
status:     FAIL
run time:   31.851 seconds

RemoteCommandError({'ssh_config': {'host': 'docker-rp-1', 'hostname': 'docker-rp-1', 'user': 'root', 'port': 22, 'password': 'UNUSED', 'identityfile': '/root/.ssh/id_rsa'}, 'hostname': 'docker-rp-1', 'ssh_hostname': 'docker-rp-1', 'user': 'root', 'externally_routable_ip': 'docker-rp-1', '_logger': <Logger rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_shadow_indexing_config-184 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f36301bd300>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f3630694b80>, '_custom_ssh_exception_checks': None}, 'host ;; communications error to 127.0.0.11#53: connection refused', 2, b'')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 182, in _do_run
    self.setup_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 260, in setup_test
    self.test.setup()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/root/tests/rptest/tests/redpanda_test.py", line 38, in setUp
    self.__redpanda.start()
  File "/root/tests/rptest/services/redpanda.py", line 2528, in start
    self.for_nodes(to_start, start_one)
  File "/root/tests/rptest/services/redpanda.py", line 1407, in for_nodes
    return list(executor.map(cb, nodes))
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/tests/rptest/services/redpanda.py", line 2520, in start_one
    self.start_node(node,
  File "/root/tests/rptest/services/redpanda.py", line 2813, in start_node
    self.write_node_conf_file(
  File "/root/tests/rptest/services/redpanda.py", line 3639, in write_node_conf_file
    fqdn = self.get_node_fqdn(node)
  File "/root/tests/rptest/services/redpanda.py", line 3607, in get_node_fqdn
    fqdn = node.account.ssh_output(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
    return method(self, *args, **kwargs)
ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-1: Command 'host ;; communications error to 127.0.0.11#53: connection refused' returned non-zero exit status 2.

JIRA Link: CORE-1873

[travisdowns]

Kind of weird, looks like the earlier dig command failed (or at least the first line out ouptut was a failure) and we pass that on to the next command. Maybe a temporary blip in DNS resolution?

I guess DNS is implemented by docker daemon here and maybe this is very early in the container lifetime? There seem to be some handwavy explanations that it may fail at this point.

Probably we can add some logging of the full output and maybe a retry.

[travisdowns]

Using "kafka" as I guess this was added for kerberos support.

[piyushredpanda]

Closing as stale.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15928

[michael-redpanda]

Automatically closing issue to match current state of CORE-1873

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15944 *https://buildkite.com/redpanda/vtools/builds/15952

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15980 *https://buildkite.com/redpanda/vtools/builds/16003 *https://buildkite.com/redpanda/vtools/builds/16016 *https://buildkite.com/redpanda/vtools/builds/16030

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.