coral icon indicating copy to clipboard operation
coral copied to clipboard

Add coral-beam for converting Calcite logical plan into Apache Beam API Java code

Open khaitranq opened this issue 4 years ago • 3 comments

End-to-end implementation and tests for translating from Pig Latin scripts into Apache Beam Java code

khaitranq avatar Nov 09 '20 23:11 khaitranq

Could you please remove Pig references and unit tests since Currently Coral supports reading from HiveQL only?

wmoustafa avatar Nov 19 '20 21:11 wmoustafa

Any update on Apache Beam integration? it will be very useful to have beam as well.

cc: @wmoustafa @khaitranq

Rep1AI avatar Jan 01 '22 09:01 Rep1AI

Any update on Apache Beam integration? it will be very useful to have beam as well.

cc: @wmoustafa @khaitranq

Thanks @RohitR1 for reaching out! Do you have a use case for this? Before merging this PR, we need to validate that the two sides of the conversion in this PR (from Pig and to Beam) commute with other sides already implemented in Coral (e.g, from Hive QL and to Trino SQL), so one can convert Hive QL (or Spark SQL) to Beam for example. It might also be best to split the PRs to two halves to ensure there is no coupling between the two conversion sides. I think this is the major remaining step here. We are not planning to pick up this now, but please feel free to make suggestions.

wmoustafa avatar Jan 05 '22 00:01 wmoustafa