Reducing coupling with kafka-journal

[grzegorz-bielski]

As you may know from kafka-flow's overview, the library is tightly coupled with kafka-journal. But in reality, only the sources from the com.evolutiongaming.kafka.flow.journal package are involved in reading the journal.

kafka-flow is already used by many internal Evo apps to a great success, and a lot of them are reading generic kafka topics instead ofkafka-journal's journal.

Direct dependency on kafka-journal brings a lot of unnecessary, transient dependencies to classpath like akka, making the library much harder to use than it should be, and the path for migration to Scala 3 unclear. In many cases kafka-journal is just treated as a source of util classes that are not strictly related to the kafka-journal domain.

My suggestion would be to move code that is shared between kafka-journal and kafka-flow into separate libraries and modules.

In particular, we can:

1. Ensure binary and source compatibility is checked:

• [x] in kafka-flow: https://github.com/evolution-gaming/kafka-flow/issues/593
• [x] in scassandra: https://github.com/evolution-gaming/scassandra/issues/250
• [x] in skafka: https://github.com/evolution-gaming/skafka/issues/420

2. kafka related

• [x] Use ConsumerRecord and ConsumerRecords from skafka directly instead of ConsRecord and ConsRecords aliases
• [ ] Extract to skafka / remove / internalize:
  • [x] PartitionOffset
  • [ ] KafkaHealthCheck
  • [x] SkafkaHelper
  • [ ] HeaderToTuple
  • [ ] TupleToHeader

3. cassandra related

• [ ] Extract to scassandra / remove / internalize:
  • [x] CassandraSession
  • [ ] CassandraHealthCheck
  • [ ] Segments
  • [x] CassandraHelper

4. Extract to separate kafka-flow-journal and kafka-flow-journal-cassandramodules:

• [ ] all *Journal* stuff with com.evolutiongaming.kafka.flow.journal from core and persistence-cassandra In the interim stage, these module could directly depend on the kafka-journal, but ideally types like SeqNr or ConsRecord would be extracted to some kafka-journal-api, which would be a shared dependency between kafka-journal and kafka-flow-journal.

5. Extract to cats-helper or other helper libs or just remove:

• [ ] RandomIdOf
• [ ] FromBytes
• [ ] ToBytes
• [ ] FromAttempt
• [ ] Fail - we could make a helper (extension) method for JournalError(a).raiseError and just use ApplicativeThrowable directly instead
• [ ] FromJsResult (play-json util)