Ben Pfaff

I expected this to be an easy fix now, with the following patch (essentially a revert): ```patch diff --git a/crates/nexmark/src/queries/q19.rs b/crates/nexmark/src/queries/q19.rs index 2ba896333..357f3d462 100644 --- a/crates/nexmark/src/queries/q19.rs +++ b/crates/nexmark/src/queries/q19.rs @@ -1,3...

Appeared again: https://github.com/feldera/feldera/actions/runs/7733079660/job/21084432540

Looking at the last ~400 failing runs, they failed with the following logs: Run 7769573622: ``` 2024-02-03T21:58:07.3625188Z +test-adapters *failed* | ---- transport::kafka::ft::test::output_test stdout ---- 2024-02-03T21:58:07.3627765Z +test-adapters *failed* | 2024-02-03 21:54:45.951101...

Categorizing these: * Runs 7639276759 and 789450558 are the same problem reaching unreachable code already described in the [initial comment](https://github.com/feldera/feldera/issues/1331#issue-2098760188). * Runs 7769573622, 7850835766, and 7850835766 all fail in the...

I split this bug into three separate ones to reflect the three sorts of causes of failure. This bug is now just for the bug identified in the initial comment.

I'm closing this until we have a reproduction case. We've done as much as we can for now.

Further reproduction at https://github.com/feldera/feldera/actions/runs/8330258471/job/22794496754?pr=1547. Relevant logs: ``` +test-adapters *failed* | failures: +test-adapters *failed* | ---- transport::kafka::ft::test::test_empty_input stdout ---- +test-adapters *failed* | 2024-03-18 16:46:12.766430 INFO librdkafka: CONFWARN [thrd:app]: No `bootstrap.servers` configured:...

We hit this again: https://github.com/feldera/feldera/actions/runs/8603451370/job/23575378448. This time it's for `transport::kafka::ft::test::proptest_kafka_end_to_end_csv_large`, which spews 10,000+ lines of diffs. Materialize hit something related: https://github.com/confluentinc/librdkafka/issues/3231. But that is about changing assignments, which we're not...

I found another one for `test_empty_input` by searching logs: https://github.com/feldera/feldera/actions/runs/8574555416/job/23501650038. Same pattern: ``` 2024-04-05T19:40:04.7119948Z +test-adapters *failed* | ---- transport::kafka::ft::test::test_empty_input stdout ---- 2024-04-05T19:40:04.7121744Z +test-adapters *failed* | 2024-04-05 19:37:50 INFO librdkafka: CONFWARN...

I think that this is a matter of essentially reverting commits 3d5f1b603 ("Implemented single GatherData struct instead of seperate channels/allocations for each worker", 2022-12-13) and ed3fc87ec ("Implemented specalized gather operator",...