tests: add assertions & logging to verifiable producer

[jcsp]

Cover letter

This is to narrow down a spooky failure seen when the consumer is apparently not waiting for a high enough offset, due to seeing a lower offset in await_consumed_offsets than the VerifiableProducer had earlier reported.

Related: https://github.com/redpanda-data/redpanda/issues/3450

Backport Required

• [x] not a bug fix
• [ ] papercut/not impactful enough to backport
• [ ] v22.2.x
• [ ] v22.1.x
• [ ] v21.11.x

UX changes

• None

Release notes

• none

[jcsp]

This ran 200x locally without failing. In CI we've seen the original failure very rarely, so it may be a while before any of these trip.

[NyaliaLui]

Hold on. Those ci failures are stranger than I thought. Looking at some briefly

[jcsp]

Those failures are highlighting a nasty behavior where VerifiableProducer can be instantiated on two nodes and their outputs will fight for updates to last_acked_offsets. That's a bug, fixed.

Unfortunately the tests that were failing mysteriously were not the ones with two producers, so the issue these assertions are hunting for is still out there.

[NyaliaLui]

It looks like some of the asserts are triggering.

@jcsp What are next steps here?

[jcsp]

Those last failures were because the logic for whether to use strict validation or not (single or multiple producers) was flipped. Fixed.

[jcsp]

@NyaliaLui please could you re-review + clear your :red_circle: if all good