Failure in `SaramaProduceTest.test_produce`

[NyaliaLui]

Version & Environment

Redpanda version: dev on CDT nightly:

https://buildkite.com/redpanda/vtools/builds/4182#01845f80-c176-4aa4-b55c-816b614deec6/6-7907

    DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\\n'")
Traceback (most recent call last):
  File "/home/ubuntu/redpanda/tests/rptest/tests/compatibility/sarama_produce_test.py", line 55, in test_produce
    subprocess.check_output(["/bin/sh", "-c", cmd],
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/bin/sh', '-c', '/opt/redpanda-tests/go/sarama/produce_test/produce_test --brokers ip-172-31-58-58:9092,ip-172-31-53-254:9092,ip-172-31-58-204:9092']' returned non-zero exit status 127.
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/compatibility/sarama_produce_test.py", line 66, in test_produce
    raise DucktapeError("sarama produce failed with " + error)
ducktape.errors.DucktapeError: sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\n'

[NyaliaLui]

again in https://buildkite.com/redpanda/vtools/builds/4196#01846250-d53f-473f-83be-eb6536cfa36f/6-7544 https://buildkite.com/redpanda/vtools/builds/4201#018463f2-bed8-437b-af13-fa9cc677ecaf/6-7604

[andijcr]

amd cdt

10 FAIL test: SaramaProduceTest.test_produce (1/18 runs) 11 failure at 2022-11-18T14:42:22.575Z: DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\n'") 12 in job https://buildkite.com/redpanda/vtools/builds/4312#018489c4-806d-40af-b364-fd8e3cd49a22

[jcsp]

https://buildkite.com/redpanda/vtools/builds/4322#01849410-c027-43f9-a277-3c890470e98f

[jcsp]

The run I just saw it in also included hangs. I'm fairly concerned if a file we bake in to the AMI is just... gone?

[jcsp]

FAIL test: SaramaProduceTest.test_produce (4/4 runs)
  failure at 2022-11-23T14:46:52.114Z: DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\\n'")
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/4361#0184a383-6120-4cfe-849f-20fd922fc70f
  failure at 2022-11-23T00:09:49.571Z: DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\\n'")
      on (?arch?, VM) in job https://buildkite.com/redpanda/vtools/builds/4352#0184a05e-e39a-4cca-9bb1-ac604e541863
  failure at 2022-11-23T03:49:41.098Z: DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\\n'")
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4354#0184a0f2-9a36-400f-ac2e-f96678585a59
  failure at 2022-11-22T15:12:22.822Z: DucktapeError("sarama produce failed with b'/bin/sh: 1: /opt/redpanda-tests/go/sarama/produce_test/produce_test: not found\\n'")
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/4345#01849e5e-c25f-42d3-9369-b80ed245db57

@NyaliaLui please can you look at this?

[jcsp]

I briefly checked this while fixing https://github.com/redpanda-data/redpanda/issues/7467 -- the test worked when run by hand on its own, and there is indeed a /opt/redpanda-tests/go/sarama/produce_test/produce_test on the test node.

[piyushredpanda]

Closing it out as it has been passing since 11/24 and failed before that.