pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Fix: Handle kafka topic not found exception in fetchPartitionCount fu…

Open Abhishek01911 opened this issue 2 months ago • 1 comments

Summary

Resolves Issue 17045

This PR fixes an issue where an exception in fetchPartitionCount, often caused by a deleted topic, would halt ingestion for an entire multi-topic table. By catching this exception, the failure is now isolated. Partitions for the deleted topic will get stuck in a "consuming" state, but ingestion from all other valid topics will continue unaffected.

Testing

Tested on a pinot table. Validated single incorrect topic doesn't stop table ingestion completely. This table contains 2 topics - adaptive-authn-gateway & adaptive-authn-gatewy. adaptive-authn-gatewy is a invalid kafka topic.

Screenshot 2025-10-27 at 10 03 23 AM Screenshot 2025-10-27 at 10 03 37 AM

Validated logs are still getting ingested for adaptive-authn-gateway. Screenshot 2025-10-28 at 4 23 06 PM

Warning message is logged for topic - adaptive-authn-gatewy Screenshot 2025-10-28 at 4 28 00 PM

Abhishek01911 avatar Oct 27 '25 17:10 Abhishek01911

Codecov Report

:x: Patch coverage is 23.07692% with 10 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 33.58%. Comparing base (51632a5) to head (9999d5e). :warning: Report is 98 commits behind head on master. :white_check_mark: All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...pache/pinot/spi/stream/StreamMetadataProvider.java 23.07% 10 Missing :warning:

:exclamation: There is a different number of reports uploaded between BASE (51632a5) and HEAD (9999d5e). Click for more details.

HEAD has 6 uploads less than BASE
Flag BASE (51632a5) HEAD (9999d5e)
unittests1 1 0
unittests 3 1
java-11 4 3
temurin 8 7
unittests2 2 1
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #17088       +/-   ##
=============================================
- Coverage     63.17%   33.58%   -29.60%     
+ Complexity     1421      726      -695     
=============================================
  Files          3104     3104               
  Lines        183247   183366      +119     
  Branches      28088    28098       +10     
=============================================
- Hits         115774    61590    -54184     
- Misses        58502   116692    +58190     
+ Partials       8971     5084     -3887     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 100.00% <ø> (+36.82%) :arrow_up:
java-21 33.58% <23.07%> (+0.09%) :arrow_up:
temurin 33.58% <23.07%> (-29.60%) :arrow_down:
unittests 33.58% <23.07%> (-29.60%) :arrow_down:
unittests1 ?
unittests2 33.58% <23.07%> (+0.09%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov-commenter avatar Nov 17 '25 23:11 codecov-commenter