Walaa Eldin Moustafa
Walaa Eldin Moustafa
Thanks @caicancai. Happy to collaborate.
Re-opening since Flink is not yet supported.
Coral to Spark SQL is fully supported. If you notice small differences in the class organization compared to what @aastha25 shared above, expect Spark module to converge to that pattern...
Trino to Coral/Rel is not yet fully supported.
> On Hive views, those seem to leverage Coral, which I don't think we want to do here. Instead of an IR layer that can be represented as a separate...
> > However, if the materialized view is stale, the method simply returns to allow SparkCatalog's loadView to run. In turn, loadView returns the metadata for the virtual view itself,...
> @wmoustafa, Read this today, was wondering if there is something we can utilize from CDC (considering iceberg has support for that) perspective ? how expensive the refreshes of a...
Yes, Trino to Spark is WIP. We will be publishing the next steps to complete the integration soon.
Currently, `coral-hive` could be used to parse Spark SQL since Hive and Spark SQL are very similar. There is an open PR (#339) for Spark SQL though. FYI @ebyhr.
Could you evaluate the Spark behavior in case Spark is used on the LHS? What triggered this issue? Is it still required? FYI @phd3.