pykafka icon indicating copy to clipboard operation
pykafka copied to clipboard

[WIP] Support RecordBatch

Open emmettbutler opened this issue 7 years ago • 2 comments

This pull request addresses https://github.com/Parsely/pykafka/issues/831 by adding support for RecordBatch message formatting to pykafka. This includes supporting the encoding and decoding of varints via the struct_helpers interface.

emmettbutler avatar Jul 26 '18 20:07 emmettbutler

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@8747398). Click here to learn what that means. The diff coverage is 75.71%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #844   +/-   ##
=========================================
  Coverage          ?   82.38%           
=========================================
  Files             ?       36           
  Lines             ?     3850           
  Branches          ?      529           
=========================================
  Hits              ?     3172           
  Misses            ?      526           
  Partials          ?      152
Impacted Files Coverage Δ
pykafka/protocol/message.py 79.87% <21.05%> (ø)
pykafka/utils/struct_helpers.py 97.95% <96.07%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8747398...7370c9d. Read the comment docs.

codecov-io avatar Jul 26 '18 20:07 codecov-io

The protocol logic for RecordBatch is implemented and untested. The next step is to figure out how to get Kafka to start speaking in RecordBatch instead of MessageSet so I can run some tests to work out the protocol implementation bugs. Once that's done, the next step will be to make changes to the producer and consumer to have them use RecordBatch when appropriate.

emmettbutler avatar Aug 15 '18 22:08 emmettbutler