Vikram Agrawal

@iamrohit, @akumarb2010 - Yes you are right. Apart from handling better cluster resources, the most important task is to manage the streaming pipeline. One of the key metrics to consider...

@peterburnash - Spark 3 has significant changes in Data source V2 APIs which is required for continuous streaming. We had to remove the code for continuous streaming to support the...

@adamlbailey - Looking forward to the PR.

@sap2105 - The library currently supports single stream. We need to make changes to handle multiple streams. Would you be interested in working on it and raising a PR?

@HariprasadAllaka1612 - You need to check the executor logs to see why the checkpoint location is not created.

Sure. Let me reproduce the issue and provide a fix.

Hey @HariprasadAllaka1612 - I could not spend any cycle on it. Let me do it tomorrow.

@HariprasadAllaka1612 - I can reproduce the issue in certain scenarios. I still haven't got to the bottom of it. Will start a PR as soon as we have a solution...

@HariprasadAllaka1612 and @3mlabs - Apologies. I am a little behind on this one. This looks like an involved issue and we have to understand how forEachBatch is working and why...

assigning the issue to @abhishekd0907