dinky icon indicating copy to clipboard operation
dinky copied to clipboard

[Bug] [Devops] When using Dinky dev version ,data sources that support HDFS to manage paimon

Open dsanww opened this issue 1 year ago • 4 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

When using Dinky dev version(commit:114cc18179bb857c93c34324b32438f97ef8af1c) ,data sources that support HDFS to manage paimon An error was reported while submitting the paimon task: Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.paimon.utils.HadoopUtils.readHadoopXml(HadoopUtils.java:220) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.utils.HadoopUtils.addHadoopConfIfFound(HadoopUtils.java:175) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.utils.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:113) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.catalog.CatalogContext.(CatalogContext.java:53) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:73) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:81) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionBase.initPaimonCatalog(ActionBase.java:78) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionBase.(ActionBase.java:65) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.(SynchronizationActionBase.java:77) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.(SyncDatabaseActionBase.java:68) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.(MySqlSyncDatabaseAction.java:109) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:52) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:31) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SynchronizationActionFactoryBase.create(SynchronizationActionFactoryBase.java:45) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SyncDatabaseActionFactoryBase.create(SyncDatabaseActionFactoryBase.java:47) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:87) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:43) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:145) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.18.0] at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:118) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:84) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.flinksql.Submitter.executeJarJob(Submitter.java:263) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.flinksql.Submitter.submit(Submitter.java:142) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.MainApp.main(MainApp.java:53) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0]

What you expected to happen

no error

How to reproduce

1、Set dinky data source paimon to HDFS and specify the hdfs configuration path 2、run paimon with flink application job on dinky

Anything else

No response

Version

dev

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

dsanww avatar Oct 15 '24 02:10 dsanww

Hello @dsanww, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @dsanww, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。

github-actions[bot] avatar Oct 15 '24 02:10 github-actions[bot]

What's you sql?

aiwenmo avatar Oct 21 '24 09:10 aiwenmo

What's you sql?

paimon's DatabaseSync Action Jar

dsanww avatar Oct 21 '24 09:10 dsanww

Hello @dsanww, we need you to provide reproduction steps so that we can help you troubleshoot the problem. Issues are marked `Need Reproduce`.

你好 @dsanww, 我们需要你提供复现步骤以便于我们帮你排查问题,Issues 标注为 `Need Reproduce` 。

github-actions[bot] avatar Oct 23 '24 02:10 github-actions[bot]

Hi @, Because this issue was marked as requiring detailed reproduction steps, no response was received for 10 days. The issue is now closed. If you have any questions, please comment and reply.

Hi @, 由于该 issue 被标记为需要提供详细复现步骤,却 10 天未收到回应。现关闭 issue,若有任何问题,可评论回复。

github-actions[bot] avatar Nov 03 '24 00:11 github-actions[bot]