Support N topics written under the same directory #544

[JozoVilcek]

Problem

When using a custom partitioner, one can opt out from having topic data separated into topic specific directory. But when writing data from N topics under the same directory, the recovery of offsets is looking for files under paths which does not exists.

Solution

Detect if partitioner us adding topic name to the output path. If not, exclude it from the search path for committed files -> start search from parent directory. This however does not cover generic case when partitioner chooses some exotic path strategy.

Does this solution apply anywhere else?

• [ ] yes
• [x] no

If yes, where?

Test Strategy

Testing done:

• [x] Unit tests
• [x] Integration tests
• [ ] System tests
• [x] Manual tests

Release Plan

[ghost]

It looks like @JozoVilcek hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence. Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

[JozoVilcek]

[clabot:check]

[ghost]

@confluentinc It looks like @JozoVilcek just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

[JozoVilcek]

Does this PR make sense? Any kind of feedback is much appreciated

[JozoVilcek]

@kpatelatwork I did find you active in other MRs. Can you help me how can I bring contributions into the attention of code owners? I have problem to get feedback on this and few other old MRs

[JozoVilcek]

@janjwerner-confluent , @rishabhbits038 could you please review this request and let me know the feedback if this has a chance to get in. If yes I would work on updating the PR and make it green

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}