Add comparison docs for FsKafka vs Propulsion.Kafka

[bartelink]

Transcript of a FAQ:

hey, is there any guidance on when it makes sense to use propulsion and when it makes sense to use fskafka?

FsKafka does:

a) basic logging of errors, support for periodically emitting lag stats per the broker b) handling in Batches (BatchedConsumer.Start) - you provide a body that is not allowed to throw and takes care of any desired parallism c) handing in Batches, grouped by Key - BatchedConsumer.StartByKey https://github.com/jet/FsKafka/blob/master/src/FsKafka/ConfluentKafka.fs#L484 - it takes care of max parallelism across the groups, but if you throw, the service stops d) no throughput stats e) producers

Propulsion does:

a) handlers like b/c above but with

• more detailed logging: latencies, successes, fails
• ability to emit outcomes from handlers and compose summary stats from them cleanly
• keeps a list of tasks and manages retries
• makes reading completely async from handling (which is not a big win for Kafka but is for other things)

b) BatchedConsumer: ability to hold state of multiple streams for complex stateful consumers c) mix and match / swap ES vs CFP vs Kafka consumers with same semantics

Propulsion.Kafka

a) depends on FsKafka b) adapt Confluent.Kafka/FsKafka to ingest into Propulsion (using CK 1.x)

Reasons not to use FsKafka

Things FsKafka currently has gaps on relative to Propulsion/Propulsion.Kafka aside from scheduling capabilities / retry management:

• logging stats about batches wrt consuming or producing

Reasons to use Propulsion.Kafka

• One clear case for using Propulsion over FsKafka is when you are doing lots of parallel work that can fail or be rate limited and/or otherwise necessitate retries while maintaining good throughput - i.e. writing to Cosmos
• providing a place to hook in whitelisting/blacklisting/poison message management in a generic way

Rules of thumb

• If it can be done with FsKafka, its a damn good place to start in general - less Things. ... But if you take that too far, you're writing loads of untested snowflake infrastructure code against specific versions of client libraries with no easy way to upgrade things when breaking changes to underlying libraries come (vs Propulsion adding a binding that implements the same abstraction over a changed API contract from something like Confluent.Kafka / EventStore.ClientAPI / Microsoft.Azure.Documents etc)

[bartelink]

should mention that the AllProcessed vs OverrideWritePosition mechanism in Propulsion can be used to deal with the changelog being ahead of the reader

[bartelink]

Closing as a lot of this is covered as part of #200 (although it's not written as directly as the above)