Failed redirect for container_1482301615844_0001_01_000001

[sumitkulkarni]

Hi,

I am trying to deploy this model in my Hadoop echo system. I am using my own Apache Hadoop to deploy this presto-yarn. I have started on mandatory servers like HDFS, YARN, Zookeeper etc. And I configured the all files as per steps given in read me. But when I started the slider app I am getting following error. Following logs are printed by yarn logs -applicationId <myAppId>

2016-12-21 12:00:21,677 [AmExecutor-006] INFO state.AppState - Reviewing RoleStatus{name='COORDINATOR', key=1, desired=1, actual=0, requested=0, releasing=0, failed=0, failed recently=0, node failed=0, pre-empted=0, started=0, startFailed=0, completed=0, failureMessage=''} : expected 1 2016-12-21 12:00:21,678 [AmExecutor-006] INFO state.AppState - COORDINATOR: Asking for 1 more nodes(s) for a total of 1 2016-12-21 12:00:21,681 [AmExecutor-006] INFO state.AppState - Container ask is Capability[<memory:1500, vCores:1>]Priority[1073741825] and label = coordinator 2016-12-21 12:00:21,682 [AmExecutor-006] INFO state.AppState - Reviewing RoleStatus{name='WORKER', key=2, desired=3, actual=0, requested=0, releasing=0, failed=0, failed recently=0, node failed=0, pre-empted=0, started=0, startFailed=0, completed=0, failureMessage=''} : expected 3 2016-12-21 12:00:21,682 [AmExecutor-006] INFO state.AppState - WORKER: Asking for 3 more nodes(s) for a total of 3 2016-12-21 12:00:21,682 [AmExecutor-006] INFO state.AppState - Container ask is Capability[<memory:1500, vCores:1>]Priority[1073741826] and label = worker 2016-12-21 12:00:21,682 [AmExecutor-006] INFO state.AppState - Container ask is Capability[<memory:1500, vCores:1>]Priority[1073741826] and label = worker 2016-12-21 12:00:21,682 [AmExecutor-006] INFO state.AppState - Container ask is Capability[<memory:1500, vCores:1>]Priority[1073741826] and label = worker 2016-12-21 12:00:21,865 [AMRM Heartbeater thread] ERROR impl.AMRMClientAsyncImpl - Exception on heartbeat org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, queue=default doesn't have permission to access all labels in resource request. labelExpression of resource request=worker. Queue labels=* at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:308) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:244) at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:505) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.allocate(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, queue=default doesn't have permission to access all labels in resource request. labelExpression of resource request=worker. Queue labels=* at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:308) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:244) at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:505) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.allocate(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
... 9 more

2016-12-21 12:00:21,867 [AMRM Callback Handler Thread] INFO impl.AMRMClientAsyncImpl - Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)

[kokosing]

To me it looks like labels are misconfigured. Please make sure it works correctly before trying to use it with presto-yarn (slider). This can be helpful to test labels configuration: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/using_node_labels.html

[sumitkulkarni]

@kokosing thank you for your reply.

I have set labels in yarn but now its giving following error to me.

2016-12-22 12:18:55,985 [main] INFO appmaster.SliderAppMaster - Process has exited with exit code 0 mapped to 0 -ignoring 2016-12-22 12:18:55,985 [main] INFO workflow.WorkflowCompositeService - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED 2016-12-22 12:18:55,986 [main] INFO state.AppState - Releasing 1 containers 2016-12-22 12:18:55,986 [main] INFO appmaster.SliderAppMaster - Application completed. Signalling finish to RM 2016-12-22 12:18:55,986 [main] INFO appmaster.SliderAppMaster - Unregistering AM status=FAILED message=AMRMClientAsync.onError() received org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invailid resource request, queue=default specified node label expression in a resource request has resource name = /default-rack at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:289) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:244) at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:505) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

[kokosing]

I have set labels in yarn but now its giving following error to me.

How do you know it is configured properly (how do you tested that)? Can you run other YARN application which is using this label and check it uses proper resources in YARN dashboard?

To me it still looks like something is misconfigured regarding labels and it has nothing related to slider (or presto-yarn).