[Bug] hive 脚本 执行 失败 async get EngineNode AMErrorException: errCode: 30001 ,desc: Failed to createEngine ask em ServiceInstance

[litongqian]

trafficstars

Search before asking

• [X] I searched the issues and found no similar issues.

Linkis Component

linkis-cg-entrance, linkis-cg-engineConnManager

What happened + What you expected to happen

执行 sh bin/linkis-cli -submitUser hadoop -engineType hive-3.0.0 -codeType hql -code "show tables" 发生错误:

==> linkis-cg-engineconnmanager.log <==
746-2022-08-07 17:15:01.976 [INFO ] [qtp456228477-22                         ] o.a.l.e.s.s.i.LinuxProcessEngineConnLaunchService (41) [info] - Try to launch a new EngineConn with CommonProcessEngineConnLaunchRequest(572e577b-9136-4b4e-94d2-60199f87f7ea,hadoop,[[key: userCreator, value: {"creator":"LINKISCLI","user":"hadoop"}, str: hadoop-LINKISCLI], [key: engineType, value: {"engineType":"hive","version":"3.0.0"}, str: hive-3.0.0]],org.apache.linkis.manager.engineplugin.common.resource.UserNodeResource@59039fd7,[org.apache.linkis.manager.common.protocol.bml.BmlResource@1c011a93, org.apache.linkis.manager.common.protocol.bml.BmlResource@10538b78],{CLASSPATH=<<LHADOOP_CONF_DIRR>><<CPS>><<LHIVE_CONF_DIRR>><<CPS>><<LPWDR>>/conf<<CPS>>/mnt/dss/LinkisInstall/conf<<CPS>><<LPWDR>>/lib/*<<CPS>>/mnt/dss/LinkisInstall/lib/linkis-commons/public-module/*<<CPS>><<LPWDR>>},[Ljava.lang.String;@2d57608a,EngineConnCreationDescImpl(dssmark_id: mark_9,null,{wds.linkis.rm.yarnqueue.memory.max=300G, wds.linkis.rm.client.core.max=10, wds.linkis.rm.instance=10, wds.linkis.rm.yarnqueue.instance.max=30, wds.linkis.rm.yarnqueue.cores.max=150, wds.linkis.rm.client.memory.max=20G, wds.linkis.rm.yarnqueue=ide}),[JarUDFLoadECMHook],[Ljava.lang.String;@68efafd0,3).
747:2022-08-07 17:15:01.987 [ERROR] [qtp456228477-22                         ] o.a.l.r.m.m.MessageExecutor (89) [executeOneMethod] - method org.apache.linkis.ecm.server.service.impl.LinuxProcessEngineConnLaunchService.launchEngineConn call failed java.lang.reflect.InvocationTargetException: null
748-	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201]
749-	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201]
750-	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]
751-	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]
752-	at org.apache.linkis.rpc.message.method.MessageExecutor.executeOneMethod(MessageExecutor.java:82) ~[linkis-rpc-1.1.1.jar:1.1.1]
--
1040-
1041-
1042-==> linkis-cg-entrance.log <==
1043-2022-08-07 17:15:02.096 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0       ] o.a.l.o.e.s.i.DefaultEngineAsyncResponseService (41) [info] - Failed to create engine bigdata03:9101_9, can retry false
1044-2022-08-07 17:15:02.371 [WARN ] [BaseTaskScheduler-Thread-39             ] o.a.l.o.e.ComputationEngineConnManager (50) [warn] - mark_9 Failed to askEngineAskRequest time taken (605)
1045:2022-08-07 17:15:02.371 [ERROR] [BaseTaskScheduler-Thread-39             ] o.a.l.o.s.a.AsyncExecTaskRunnerImpl (79) [run] - Failed to execute task astJob_9_codeExec_9 org.apache.linkis.orchestrator.ecm.exception.ECMPluginErrorException: errCode: 12003 ,desc: bigdata03:9101_9 Failed  to async get EngineNode AMErrorException: errCode: 30001 ,desc: Failed to createEngine ask em ServiceInstance(linkis-cg-engineconnmanager, bigdata03:9102) ,ip: bigdata03 ,port: 9101 ,serviceKind: linkis-cg-linkismanager ,ip: bigdata03 ,port: 9104 ,serviceKind: linkis-cg-entrance
1046-	at org.apache.linkis.orchestrator.ecm.ComputationEngineConnManager.getEngineNodeAskManager(ComputationEngineConnManager.scala:165) ~[linkis-orchestrator-ecm-plugin-1.1.1.jar:1.1.1]
1047-	at org.apache.linkis.orchestrator.ecm.ComputationEngineConnManager.askEngineConnExecutor(ComputationEngineConnManager.scala:101) ~[linkis-orchestrator-ecm-plugin-1.1.1.jar:1.1.1]
1048-	at org.apache.linkis.orchestrator.ecm.AbstractEngineConnManager.getAvailableEngineConnExecutor(EngineConnManager.scala:132) ~[linkis-orchestrator-ecm-plugin-1.1.1.jar:1.1.1]
1049-	at org.apache.linkis.orchestrator.computation.execute.DefaultCodeExecTaskExecutorManager.createExecutor(DefaultCodeExecTaskExecutorManager.scala:113) ~[linkis-computation-orchestrator-1.1.1.jar:1.1.1]
1050-	at org.apache.linkis.orchestrator.computation.execute.DefaultCodeExecTaskExecutorManager.askExecutor(DefaultCodeExecTaskExecutorManager.scala:89) ~[linkis-computation-orchestrator-1.1.1.jar:1.1.1]
--

Relevent platform

• centos7
• linkis1.1.1

Reproduction script

sh bin/linkis-cli -submitUser  hadoop  -engineType hive-3.0.0  -codeType hql  -code "show tables"

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[github-actions[bot]]

:blush: Welcome to the Apache Linkis (incubating) community!! We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

WeChat Group:

Mailing Lists:

name	description	Subscribe	Unsubscribe	archive
[email protected]	community activity information	subscribe	unsubscribe	archive

[casionone]

can you provide linkis-cg-engineconnmanager.log full detail infomation logs

Attach files by dragging & dropping, selecting or pasting them.

[litongqian]

can you provide linkis-cg-engineconnmanager.log full detail infomation logs

Attach files by dragging & dropp linkis-cg-engineconnmanager.log ing, selecting or pasting them.

linkis-cg-entrance.log

[casionone]

can you provide linkis-cg-engineconnmanager.log full detail infomation logs Attach files by dragging & dropp linkis-cg-engineconnmanager.log ing, selecting or pasting them.

linkis-cg-entrance.log

In linkis-cg-entrance.log tow things need to confirm

1. ‘No plugin found hive-3.0.1, please check your configuration’ hive plugin of version 3.0.1 is not exit, please confirm the version is right, you maybe can found some useful infomation https://linkis.apache.org/zh-CN/docs/latest/deployment/engine-conn-plugin-installation#1-%E5%BC%95%E6%93%8E%E6%8F%92%E4%BB%B6%E7%9B%AE%E5%BD%95%E7%BB%93%E6%9E%84

write log for job 2 failed org.apache.hadoop.security.AccessControlException: 
Permission denied: user=hadoop, access=EXECUTE, inode="/tmp/linkis/log":root:hdfs:drwxr-x---