hudi icon indicating copy to clipboard operation
hudi copied to clipboard

Published 20 hours ago • verified publisher icon apache

[HUDI-6297] Fixed issue in consuming transactional topic

Open ad1happy2go opened this issue 2 years ago • 4 comments

Change Logs

Fixed issue in consuming Transactional Kafka Topic.

Impact

none

Risk level (write none, low medium or high below)

none

Documentation Update

none

Contributor's checklist

  • [ ] Read through contributor's guide
  • [ ] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

ad1happy2go avatar Jun 27 '23 10:06 ad1happy2go

CI report:

  • 9395a5cf8b4309e028959d0da471cbb66239e80b Azure: FAILURE
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Jun 27 '23 14:06 hudi-bot

@ad1happy2go : Can you introduce the change in KafkaSource (with a config) which can wrap the RDD and handle the behavior instead of making the change in DeltaSync ?

bvaradar avatar Oct 30 '23 21:10 bvaradar

hey @bvaradar : can you help with this issue. Do you have pointers on exactly once from kafka. looks like reading it OOB might result in data consistency issues as reported above.

nsivabalan avatar Nov 15 '23 20:11 nsivabalan

Hey @nsivabalan , @bvaradar , @ad1happy2go - is this absolutely not going to happen anytime soon? It's preventing us from directly ingesting a large majority of our Kafka topics in our organisation and i'm very surprised it's not a more widely experienced issue, given its a common feature of topics produced through kafka-streams applications

danielford831 avatar Apr 10 '24 09:04 danielford831