incubator-heron icon indicating copy to clipboard operation
incubator-heron copied to clipboard

Published 20 hours ago • verified publisher icon apache

Using yarn scheduler, transformsTopology and WindowedWordCountTopology can not be run well.

Open dttlgotv opened this issue 4 years ago • 1 comments

I run SmartWatchTopology example well on yarn well, two yarn containers are created and topology can work well. But when I run TransformsTopology on yarn, only one yarn container is created ,so topology can not work. Comparing the driver.err files, when TransformsTopology is running, less logs can not be seen than SmartWatchTopology such as :

localHostTrusted = false, remoteHostTrusted = false 2020-03-16 15:19:01,926 INFO impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1584341343088_0001_01_000002 2020-03-16 15:19:01,981 INFO impl.NMClientAsyncImpl: Processing Event EventType: QUERY_CONTAINER for Container container_1584341343088_0001_01_000002 2020-03-16 15:21:13,302 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2020-03-16 15:21:13,305 INFO zookeeper.ZooKeeper: Session: 0x1000000f3f50003 closed 2020-03-16 15:21:13,305 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x1000000f3f50003 2020-03-16 15:21:13,393 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

heron submit yarn TransformsTopology part of driver.err: 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/folders/zr/dgwn7zgn3fbcjmv25t_ybsc80000gp/T/ 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac OS X 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:os.arch=x86_64 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:os.version=10.14.6 2020-03-16 15:31:21,116 INFO zookeeper.ZooKeeper: Client environment:user.name=guoxinghua 2020-03-16 15:31:21,117 INFO zookeeper.ZooKeeper: Client environment:user.home=/Users/guoxinghua 2020-03-16 15:31:21,117 INFO zookeeper.ZooKeeper: Client environment:user.dir=/Users/guoxinghua/hadoop-3.2.1/hdfs_dir/nm-local-dir/usercache/guoxinghua/appcache/application_1584341343088_0004/container_1584341343088_0004_01_000001 2020-03-16 15:31:21,117 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@39e43310 2020-03-16 15:31:21,128 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-03-16 15:31:21,129 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 2020-03-16 15:31:21,135 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1000000f3f50020, negotiated timeout = 30000 2020-03-16 15:31:21,140 INFO state.ConnectionStateManager: State change: CONNECTED

heron submit yarn SmartWatchTopology part of driver.err: 2020-03-16 15:18:47,230 INFO state.ConnectionStateManager: State change: CONNECTED 2020-03-16 15:19:01,066 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-16 15:19:01,438 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-16 15:19:01,777 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-16 15:19:01,879 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-03-16 15:19:01,926 INFO impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1584341343088_0001_01_000002 2020-03-16 15:19:01,981 INFO impl.NMClientAsyncImpl: Processing Event EventType: QUERY_CONTAINER for Container container_1584341343088_0001_01_000002 2020-03-16 15:21:13,302 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2020-03-16 15:21:13,305 INFO zookeeper.ZooKeeper: Session: 0x1000000f3f50003 closed 2020-03-16 15:21:13,305 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x1000000f3f50003 2020-03-16 15:21:13,393 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

dttlgotv avatar Mar 16 '20 08:03 dttlgotv

Hmm. You mean other topologies work but these two don't?

nwangtw avatar Mar 19 '20 04:03 nwangtw