kafka-connect-hdfs icon indicating copy to clipboard operation
kafka-connect-hdfs copied to clipboard

Recover the offset from HDFS, even if topic name is not present in storage path

Open hariprasad-k opened this issue 3 years ago • 3 comments

Problem

Exactly-once semantics should also work without topic name included in the path.

Solution

The function that recover offsets from file names will use the correct path by taking into account a new configuration flag path_include_topicname introduced with confluentinc/kafka-connect-storage-common#126.

Further, the backward compatibility of appending the topic name is retained by default value true for path_include_topicname.

Does this solution apply anywhere else?
  • [x] yes
  • [ ] no
If yes, where?

This change is necessary to retain EOS semantics to support flexible storage partitioning scheme as proposed here: confluentinc/kafka-connect-storage-common#126

Test Strategy

Testing done:
  • [x] Unit tests
  • [x] Integration tests
  • [ ] System tests
  • [ ] Manual tests

Release Plan

hariprasad-k avatar Nov 03 '20 17:11 hariprasad-k

@confluentinc It looks like @hariprasad-k just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

ghost avatar Nov 03 '20 17:11 ghost

Fixes #515

hariprasad-k avatar Nov 03 '20 17:11 hariprasad-k

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

cla-assistant[bot] avatar Aug 27 '23 12:08 cla-assistant[bot]