kafka-connect-hdfs
kafka-connect-hdfs copied to clipboard
Recover the offset from HDFS, even if topic name is not present in storage path
Problem
Exactly-once semantics should also work without topic name included in the path.
Solution
The function that recover offsets from file names will use the correct path by taking into account a new configuration flag path_include_topicname
introduced with confluentinc/kafka-connect-storage-common#126.
Further, the backward compatibility of appending the topic name is retained by default value true
for path_include_topicname
.
Does this solution apply anywhere else?
- [x] yes
- [ ] no
If yes, where?
This change is necessary to retain EOS semantics to support flexible storage partitioning scheme as proposed here: confluentinc/kafka-connect-storage-common#126
Test Strategy
Testing done:
- [x] Unit tests
- [x] Integration tests
- [ ] System tests
- [ ] Manual tests
Release Plan
@confluentinc It looks like @hariprasad-k just signed our Contributor License Agreement. :+1:
Always at your service,
clabot
Fixes #515
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.