seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Bug][flink-runtime][connectors-v2] Flink register table Environment The running mode is set to`job.mode`

Open zhilinli123 opened this issue 2 years ago • 4 comments

issues: #4809

Purpose of this pull request

Check list

  • [ ] Code changed are covered with tests, or it does not need tests for reason:
  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

zhilinli123 avatar May 25 '23 02:05 zhilinli123

hive Source:

create table  test3 (
  name string,
  age int,
  emails ARRAY<STRING>,
  phones map<STRING, string>
);
INSERT INTO TABLE test3 
VALUES ('list',12,array("[email protected]", "[email protected]", "[email protected]"), map('home', "13987167117", 'office', "15178968888"));

hive Sink:

create table  test3_sink (
  name string,
  age int,
  emails ARRAY<STRING>,
  phones map<STRING, string>
);

Conf :
env {
  # You can set spark configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}



source {
  Hive {
    table_name = "default.test3"
    metastore_uri = "thrift://localhost:9083"
    result_table_name = "user"
  }
}


transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
  Replace {
    source_table_name = "user"
    result_table_name = "user_1"
    replace_field = "name"
    pattern = "s"
    replacement = "w"
    is_regex = true
    replace_first=true
  }
}

sink {
# Console{ source_table_name = "user" }
Hive {
    source_table_name = "user_1"
    table_name = "default.test3_sink"
    metastore_uri = "thrift://localhost:9083"
 }
}


table result test3_sink :
liwt    12      ["[email protected]","[email protected]","[email protected]"]   {"office":"15178968888","home":"13987167117"}

Class:org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

image

zhilinli123 avatar May 25 '23 02:05 zhilinli123

image Rerun CI?

zhilinli123 avatar May 25 '23 03:05 zhilinli123

image Ci rerun

zhilinli123 avatar Jun 05 '23 08:06 zhilinli123

PTAL: @TyrantLucifer @TaoZex

zhilinli123 avatar Jul 31 '23 07:07 zhilinli123

Waiting for the e2e issue to be fixed.

ic4y avatar Aug 05 '23 12:08 ic4y

Waiting for the e2e issue to be fixed.

Done

zhilinli123 avatar Aug 06 '23 14:08 zhilinli123

ci run @ic4y @liugddx

zhilinli123 avatar Aug 08 '23 11:08 zhilinli123

@TyrantLucifer PTAL

ic4y avatar Aug 10 '23 10:08 ic4y