dl-on-flink
dl-on-flink copied to clipboard
运行Linear.java python报dl_on_flink_framework依赖不存在
我执行的步骤
按照初始化步骤对dl-on-flink进行打包 新建了一个服务用来跑flink任务 服务中引入dl-on-flink-dist
<dependency>
<groupId>org.flinkextended</groupId>
<artifactId>dl-on-flink-dist</artifactId>
<version>0.6.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-python_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
复制dl-on-flink-examples-tensorflow项目下的Linear.java到新服务中运行 对应报错信息
[08-29 17:52:28] [INFO] [org.mortbay.log:67] Start logging process worker:0 with org.flinkextended.flink.ml.cluster.node.runner.python.log.StdOutErrorProcessOutputConsumer@610546a2
[08-29 17:52:28] [INFO] [org.mortbay.log:67] Start logging process worker:1 with org.flinkextended.flink.ml.cluster.node.runner.python.log.StdOutErrorProcessOutputConsumer@4ea7c222
Traceback (most recent call last):
File "/Users/lsz/code/AppAIRealTimeDataProvider/startup.py", line 22, in <module>
Traceback (most recent call last):
File "/Users/lsz/code/AppAIRealTimeDataProvider/startup.py", line 22, in <module>
from dl_on_flink_framework import context
ModuleNotFoundError: No module named 'dl_on_flink_framework'
from dl_on_flink_framework import context
ModuleNotFoundError: No module named 'dl_on_flink_framework'
[08-29 17:52:29] [ERROR] [org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner:262] Got exception during python running
org.flinkextended.flink.ml.util.MLException: worker:0 python process exited with code 1
at org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner.runProcess(ProcessPythonRunner.java:145)
at org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner.runScript(ProcessPythonRunner.java:111)
at org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner.runScript(CommonMLRunner.java:276)
at org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner.run(CommonMLRunner.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[08-29 17:52:29] [ERROR] [org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner:262] Got exception during python running
org.flinkextended.flink.ml.util.MLException: worker:1 python process exited with code 1
at org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner.runProcess(ProcessPythonRunner.java:145)
at org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner.runScript(ProcessPythonRunner.java:111)
at org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner.runScript(CommonMLRunner.java:276)
at org.flinkextended.flink.ml.cluster.node.runner.CommonMLRunner.run(CommonMLRunner.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.AbstractScriptRunner:41] worker:0 mark input queue finished.
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.AbstractScriptRunner:41] worker:1 mark input queue finished.
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner:248] Python scriptRunner for worker:1 closed
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.python.ProcessPythonRunner:248] Python scriptRunner for worker:0 closed
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.NodeHeartBeatRunner:86] worker:1 heartbeat thread interrupted
[08-29 17:52:29] [INFO] [org.flinkextended.flink.ml.cluster.node.runner.NodeHeartBeatRunner:86] worker:0 heartbeat thread interrupted