auto_commit is not failure safe

[FlorianLudwig]

When processing an entry fails auto_commit will mark message as processed including the failed one.

Simple example:

async for msg in consumer:
  1/0

Does consumes one message while it should have consumed none.

One possible solution would be to mark a message for auto_commit on __anext__ on the iteration after the respective item.

[tvoinarovskyi]

Hmm, that is true, but it is quite hard to determine the border here. Explicit commit is probably the most reliable way to handle this case. If we make the auto-commit run on next iteration, should getmany also only auto-commit on next iteration?

[shargan]

One possible solution would be to mark a message for auto_commit on __anext__ on the iteration after the respective item.

def next_message_that_matters(consumer, match):
    async for msg in consumer:
        if msg.key == match:
            return msg

If the consumer only marked a message as read on the subsequent __anext__ invocation, this would simply re-consume the same message over and over.

Auto-commit, by its very nature, doesn't take into account post-consumption processing. You may be better served by implementing your own offset commit logic. Alternatively, you could seek back to the desired position during an exception.