[Bug]: Flink Sql field mapping error in Mixed Format

[shendanfengg]

What happened?

When the order of my Flink Sql Schema is not the same as the Mixed-Format table's Schema, Flink will write the wrong data. Specific sql examples can be found in the How to reproduce module. We can clearly see that the age and name columns are misaligned.

Affects Versions

master

What engines are you seeing the problem on?

Flink

How to reproduce

1. create a mixed-hive table create table db_name.table_name ( id string, name string, age string, primary key (id) ) using arctic;

2. Write Flink sql

CREATE TABLE datagen_source ( id INT, age INT, name STRING ) WITH ( 'connector' = 'datagen', 'fields.id.kind' = 'sequence', 'fields.id.start' = '1', 'fields.id.end' = '100', 'fields.name.constant' = 'bob', 'fields.age.kind' = 'sequence', 'fields.age.start' = '1', 'fields.age.end' = '100' );

create catalog hive_catalog with( 'type'='arctic', 'metastore.url'='thrift://xxxx:xxxx/catalog_name' );

insert into hive_catalog.db_name.table_name /*+ OPTIONS('amoro.emit.mode'='file') */ SELECT CAST(id AS STRING) as id, CAST(age AS STRING) as age, name as name FROM datagen_source;

3.Run select * on the Terminal page, and then we can see that the age and name columns are reversed.

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[czy006]

@shendanfengg Can you tell me which Amoro version and Flink version you are using? Or there are some specific trigger cases

[shendanfengg]

@shendanfengg Can you tell me which Amoro version and Flink version you are using? Or there are some specific trigger cases

I'm running this with Amoro version 0.5.x and Flink version 1.14, but given that the core Flink code (the column mapping part) hasn't changed much between 0.5.x and 0.6.x, I'm assuming that this issue will occur in either Flink version

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'