John Spray

@abhijat can you add the link to the build where it failed if it was in CI? Or perhaps this was clustered ducktape run by hand... I saw this log...

4 failures on arm: failure at 2022-11-21T07:40:44.803Z: in job https://buildkite.com/redpanda/redpanda/builds/18848#018498a2-1792-4269-af80-9d74661e8a43 failure at 2022-11-21T08:01:31.032Z: in job https://buildkite.com/redpanda/redpanda/builds/18850#018498b4-8859-4f40-a46b-cbb4c2774912 failure at 2022-11-20T07:52:52.860Z: in job https://buildkite.com/redpanda/redpanda/builds/18842#01849385-ef55-4d3b-aa1b-bd7d84e660e8 failure at 2022-11-19T07:28:36.045Z: in job https://buildkite.com/redpanda/redpanda/builds/18826#01848e44-e793-46fb-aa24-3364f6c3d519

FAIL test: ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True (2/47 runs) failure at 2022-11-23T07:42:59.801Z: on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/18961#0184a2ee-dda7-4e64-adb8-e53c6f012e3b failure at 2022-11-23T07:31:49.734Z: on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/18960#0184a2e5-b302-40c3-bf26-467c83070c18

fixing here https://github.com/redpanda-data/redpanda/pull/7471

FAIL test: EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True (2/50 runs) failure at 2022-11-21T07:41:53.231Z: TypeError('cannot unpack non-iterable NoneType object') in job https://buildkite.com/redpanda/redpanda/builds/18848#018498a2-1793-4ba8-af2f-35a508bfe19f failure at 2022-11-19T07:42:30.317Z: TypeError('cannot unpack non-iterable NoneType object') in job https://buildkite.com/redpanda/redpanda/builds/18828#01848e55-6349-4a70-a749-c9776019b3ac

Analyzing this failure: https://buildkite.com/redpanda/redpanda/builds/19009#0184a698-b536-4473-be39-21e5bc9c8628 This looks like a hang in remote partition read path. It is busy spinning like this: ``` DEBUG 2022-11-23 23:33:17,579 [shard 1] cloud_storage - [fiber18 kafka/topic-ynmjqwyivv/0]...

So the reader is asking for kafka offset 7724, which corresponds to a tx_fench batch. We're finding the offset, but skipping it because we use a raft_data filter, and then...

FAIL test: ShadowIndexingTxTest.test_shadow_indexing_aborted_txs (2/21 runs) failure at 2022-11-25T08:01:45.162Z: AssertionError("produced and consumed messages differ, produced length: 13875, consumed length: 1162, first mismatch: produced: b'1258', consumed: None (offset: -1)") on (arm64, container)...

This is still live today, after the merge of #7366 https://buildkite.com/redpanda/redpanda/builds/19621#018504dc-4135-4db7-968d-778a14c6c14f

nit: please can you use the `component: foo bar` style for commit message first lines, makes it much easier to browse history later