[Feature][starter] support user define parameter on spark/flink engine

[liunaijie]

Purpose of this pull request

close #6376

Does this PR introduce any user-facing change?

No

How was this patch tested?

add example to test.

Check list

• [x] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
• [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• [x] If you are contributing the connector code, please check that the following files are updated:
  1. Update change log that in connector document. For more details you can refer to connector-v2
  2. Update plugin-mapping.properties and add new connector information in it
  3. Update the pom file of seatunnel-dist
• [ ] Update the release-note.

[liunaijie]

@Hisoka-X PTAL

[Hisoka-X]

How about add some intro in https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/docs/en/concept/config.md? Then user can easiler to found this feature.

[liunaijie]

How about add some intro in https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/docs/en/concept/config.md? Then user can easiler to found this feature.

Sure, will do it.

[Hisoka-X]

Could the processing parameter logic of different engines be unified together? It seems that zeta's processing code is a bit scattered now.

https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java#L130

https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/args/ClientCommandArgs.java#L119

How about this one?

[liunaijie]

Could the processing parameter logic of different engines be unified together? It seems that zeta's processing code is a bit scattered now.

https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java#L130

https://github.com/apache/seatunnel/blob/e5df7f00f8e21edcc5168c4f3dedfc7f85a4a150/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/args/ClientCommandArgs.java#L119

How about this one?

Done, PTAL

[Hisoka-X]

Something not right.

[liunaijie]

Something not right.

yes i see this error. i have some test code in other branch. in my local desktop (windows) this ut can run successful, so run some test by github action.

[liunaijie]

Something not right.

I has some test, find in github action windows test, it can't read the user variables system properties. But in my local window desktop, it can read the properties, so i am disable this test in window to pass to ci.

the screenshot in my local windows desktop

[Hisoka-X]

Thanks @liunaijie for updated. But I think we should find the reason instead of disable it.

[hailin0]

Thanks @liunaijie for updated. But I think we should find the reason instead of disable it.

+1

@liunaijie

[liunaijie]

@Hisoka-X @hailin0 Done, PTAL. When call ConfigFactory.systemProperties() method, it has an static variable to hold the system properties. so it will only read system properties once. if the properties changed, it can't read the new properties.

[liunaijie]

@Hisoka-X @hailin0 PTAL