Feature: Consider supporting flink sql

[caicancai]

Whether to consider supporting flink sql

[wmoustafa]

Yes, it is in the roadmap, but as far as I know no one is actively working on it at the moment. Do you have a use case in mind?

[wmoustafa]

Since Flink supports streaming, you might check out Coral's incremental view maintenance features as well. It complements streaming on the batch side.

[wmoustafa]

FYI @aastha25 @ljfgem @KevinGe00 @yiqiangin

[caicancai]

Thank you for your answer. I will try this part of the work myself first. If there is any progress, I will be happy to contribute PR.

[wmoustafa]

Are you thinking of Flink SQL as a dialect on the input or output side? (or which one will you focus on)?

[caicancai]

Strictly speaking, I don’t think this is a conversion between dialects. I think it is a conversion between models. If it is just a conversion between dialects, why not use ANTLR to extract key information to spell SQL (if you do this, obviously this is a trap)

[caicancai]

你是否认为 Flink SQL 是输入端或输出端的一种方言？（或者你会关注哪一个）？

I have tried the conversion of spark sql before, and I am still learning, and I am more concerned about spark sql and flink sql

[wmoustafa]

Sorry should have clarified my question. When you said you would consider implementing Flink SQL integration with Coral, did you refer to integration on the input side (Flink SQL to Coral) or the output side (Coral to Flink SQL)?

[caicancai]

sink

[wmoustafa]

As in (Coral to Flink SQL)?

[caicancai]

如（珊瑚到Flink SQL）？

yes

[caicancai]

What I thought before was that maybe it would be easier to use a standard SQL method to convert to any other SQL dialect. But obviously your goals are much bigger

[caicancai]

I hope I can make a contribution to the project in the future, because this is what I wanted to do before

[wmoustafa]

Thanks @caicancai. Happy to collaborate.

[caicancai]

Thank you for your patient reply

[aastha25]

@caicancai thank you for picking this line of work. Please refer to the provided sample pseudo-code below to structure your code:

Convert(RelNode coralRelNode) {
          // Step 1: Convert CoralRel IR to CoralSqlNode
          CoralRelToSqlNodeConverter rel2sql = new CoralRelToSqlNodeConverter();
          SqlNode coralSqlNode = rel2sql.convert(coralRelNode);

         // Step 2: Apply Coral to Flink SQL transformations using a custom SqlShuttle
          SqlNode flinkSqlNode = coralSqlNode.accept(new CoralToFlinkSqlNodeConverter())

         // Step 3: Convert the resulting Flink SQL to a string
          String flinkSQL = flinkSqlNode.toSqlString();
}

In this code:

1. CoralRelToSqlNodeConverter is an existing class in Coral that converts Coral IR represented as a RelNode to its SqlNode representation.
2. To implement the CoralToFlinkSqlNodeConverter, please use DataTypeDerivedSqlCallConverter and CoralToTrinoSqlCallConverter as a reference. Each transformer in the SqlShuttle for this converter should extend SqlCallTransformer to modify the SqlCalls in the input.

By following this structure, you can effectively model your code for converting Coral IR to Flink SQL.

[wmoustafa]

Re-opening since Flink is not yet supported.

[caicancai]

@aastha25 Thanks for the tip, I'll try it

[caicancai]

I'm going to start trying this part out

[caicancai]

I want to ask if sink is fully supported by spark sql.

[wmoustafa]

Coral to Spark SQL is fully supported. If you notice small differences in the class organization compared to what @aastha25 shared above, expect Spark module to converge to that pattern (which @aastha25 shared) soon.

[caicancai]

@wmoustafa Sorry, I was a little busy a while ago, I will start to try now. I want to put this idea into practice