kinesis-sql icon indicating copy to clipboard operation
kinesis-sql copied to clipboard

Published 20 hours ago • verified publisher icon qubole

Migrate to DataSource V2 APIs

Open itsvikramagr opened this issue 7 years ago • 8 comments
trafficstars

Current work is developed against SPARK 2.2.X. There are new data-source APIs in SPARK-2.3.0. We need to migrate to those APIs

itsvikramagr avatar Mar 09 '18 11:03 itsvikramagr

Is there a roadmap or plan for this?

krish7919 avatar Apr 09 '18 13:04 krish7919

@krish7919 - We plan to complete this in next 2-3 months. if you are interested to pick it up, I would be happy to help you with design discussions and PR reviews.

itsvikramagr avatar Apr 10 '18 07:04 itsvikramagr

@itsvikramagr - anything i could help with, while porting to DataSource V2 - e.g Dynamodb version of MetadataCommitter ?

saj1th avatar May 22 '18 22:05 saj1th

@saj1th - Thanks for your interest. DyanamoDb version of MetadataCommitter will anyway be useful with current data source APIs. Please go ahead and raise an issue and PR for the same.

itsvikramagr avatar May 25 '18 10:05 itsvikramagr

@itsvikramagr - Couldn't find changes related to DataSourceV2 / ContinuousReadSupport in 2.3.0 branch. Is it still a work in progress ?

saj1th avatar May 25 '18 19:05 saj1th

@saj1th - 2.3.0 branch currently has support for micro-batch execution. Migrating it in DatasourceV2 and adding support for ContinousRead will take some time.

itsvikramagr avatar May 28 '18 05:05 itsvikramagr

I understand the current version was tested against Spark 2.2.X, but does it also run on Spark 2.3?

rodmaz avatar Aug 29 '18 21:08 rodmaz

Hey @rodmaz - I missed your message. It will run on spark 2.3. Current master is on 2.3.1

itsvikramagr avatar Sep 10 '18 16:09 itsvikramagr