Got UploaderException when submit topology on kubernetes

[sebastienpattyn93]

trafficstars

I have a Kubernetes cluster with 3 nodes where Heron is deployed. I am able to visit the heron ui and to curl the heron api-server.

When I try to submit a topology I keep getting the same error:

heron submit kubernetes --verbose  ./dist/word-count-topology.pex   -   Word_Count_Topology 
[2018-06-07 08:08:26 +0000] [DEBUG]: Input Command Line Args: {'config_property': [], 'dry_run_format': 'colored_table', 'topology-file-name': './dist/word-count-topology.pex', 'cluster/[role]/[env]': 'kubernetes', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 'subcommand': 'submit', 'deploy_deactivated': False, 'topology_main_jvm_property': [], 'extra_launch_classpath': '', 'service_url': '', 'topology-class-name': '-', 'verbose': 'True'}
[2018-06-07 08:08:26 +0000] [DEBUG]: Input Command Line Args: {'config_property': [], 'dry_run_format': 'colored_table', 'topology-file-name': './dist/word-count-topology.pex', 'cluster/[role]/[env]': 'kubernetes', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 'subcommand': 'submit', 'deploy_deactivated': False, 'topology_main_jvm_property': [], 'extra_launch_classpath': '', 'service_url': '', 'topology-class-name': '-', 'verbose': 'True'}
[2018-06-07 08:08:26 +0000] [DEBUG]: Using cluster definition from file /home/ubuntu/.config/heron/kubernetes/cli.yaml
[2018-06-07 08:08:26 +0000] [DEBUG]: Processed Command Line Args: {'config_property': [], 'topology-file-name': './dist/word-count-topology.pex', 'verbose': 'True', 'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'ubuntu', 'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 'role': 'ubuntu', 'dry_run_format': 'colored_table', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 'topology_main_jvm_property': [], 'environ': 'default', 'service_url': 'http://10.128.0.20:32674', 'topology-class-name': '-'}
[2018-06-07 08:08:26 +0000] [DEBUG]: Submit Args {'config_property': [], 'topology-file-name': './dist/word-count-topology.pex', 'verbose': 'True', 'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'ubuntu', 'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 'role': 'ubuntu', 'dry_run_format': 'colored_table', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 'topology_main_jvm_property': [], 'environ': 'default', 'service_url': 'http://10.128.0.20:32674', 'topology-class-name': '-'}
[2018-06-07 08:08:26 +0000] [DEBUG]: Importing - from ./dist/word-count-topology.pex
[2018-06-07 08:08:26 +0000] [DEBUG]: Invoking class using command: ``./dist/word-count-topology.pex Word_Count_Topology''
[2018-06-07 08:08:26 +0000] [DEBUG]: Heron options: {cmdline.topologydefn.tmpdirectory=/tmp/tmpW58aFW,cmdline.topology.initial.state=RUNNING}
[2018-06-07 08:08:27 +0000] [INFO]: Launching topology: 'Word_Count_Topology'
[2018-06-07 08:08:27 +0000] [INFO]: {'config_property': [], 'topology-file-name': './dist/word-count-topology.pex', 'verbose': 'True', 'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'ubuntu', 'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 'role': 'ubuntu', 'dry_run_format': 'colored_table', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 'topology_main_jvm_property': [], 'environ': 'default', 'service_url': 'http://10.128.0.20:32674', 'topology-class-name': '-'}
[2018-06-07 08:08:27 +0000] [DEBUG]: Starting new HTTP connection (1): 10.128.0.20
[2018-06-07 08:08:27 +0000] [DEBUG]: http://10.128.0.20:32674 "POST /api/v1/topologies HTTP/1.1" 500 94
[2018-06-07 08:08:27 +0000] [ERROR]: Failed to instantiate uploader class 'org.apache.heron.uploader.dlog.DLUploader'
[2018-06-07 08:08:27 +0000] [ERROR]: Failed to launch topology 'Word_Count_Topology' 
[2018-06-07 08:08:27 +0000] [DEBUG]: Elapsed time: 0.409s.

Looking in my Kubernetes POD logs I found the following error:

 [2018-06-07 08:08:27 +0000] [SEVERE] com.twitter.heron.apiserver.resources.TopologyResource: error submitting topology Word_Count_Topology 
com.twitter.heron.spi.uploader.UploaderException: Failed to instantiate uploader class 'org.apache.heron.uploader.dlog.DLUploader'
	at com.twitter.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:411)
	at com.twitter.heron.apiserver.actions.SubmitTopologyAction.execute(SubmitTopologyAction.java:33)
	at com.twitter.heron.apiserver.resources.TopologyResource.submit(TopologyResource.java:223)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) 
 	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:564)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.heron.uploader.dlog.DLUploader
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at com.twitter.heron.spi.utils.ReflectionUtils.newInstance(ReflectionUtils.java:34)
	at com.twitter.heron.spi.utils.ReflectionUtils.newInstance(ReflectionUtils.java:25)
	at com.twitter.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:408)
	... 51 more 

I tried with other example topologies from heron but got the same error.

[kramasamy]

@cckellogg - can you check this?

[cckellogg]

@sebastienpattyn93 what version for heron are you running? What is the yaml for apiserver deployment? It looks like the heron-apiserver is configured wrong. I don't think there has been a release since the packaging has changed. This class does not exist org.apache.heron.uploader.dlog.DLUploader. If you change the org.apache to com.twitter it should work.

[sebastienpattyn93]

Hi @cckellogg I'm using the yaml from this [turotial] (https://apache.github.io/incubator-heron/docs/operators/deployment/schedulers/kubernetes/#general-kubernetes-clusters)

#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing,
#  software distributed under the License is distributed on an
#  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
#  KIND, either express or implied.  See the License for the
#  specific language governing permissions and limitations
#  under the License.

##
## Heron API server deployment
##
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: heron-apiserver
  name: heron-apiserver
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: heron-apiserver
  labels:
    app: heron-apiserver
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: heron-apiserver
  namespace: default

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heron-apiserver
  labels:
    app: heron-apiserver
  namespace: default
spec:
  selector:
    matchLabels:
      app: heron-apiserver
  replicas: 1
  template:
    metadata:
      labels:
        app: heron-apiserver
    spec:
      serviceAccountName: heron-apiserver
      terminationGracePeriodSeconds: 0
      tolerations:
        - key: "node.kubernetes.io/not-ready"
          operator: "Equal"
          effect: "NoExecute"
          tolerationSeconds: 10
        - key: "node.alpha.kubernetes.io/notReady"
          operator: "Equal"
          effect: "NoExecute"
          tolerationSeconds: 10
        - key: "node.alpha.kubernetes.io/unreachable"
          operator: "Equal"
          effect: "NoExecute"
          tolerationSeconds: 10
      containers:
        - name: heron-apiserver
          image: heron/heron:latest
          command: ["sh", "-c"]
          args:
            - >-
              heron-apiserver
              --base-template kubernetes
              --cluster kubernetes
              -D heron.statemgr.connection.string=zookeeper:2181
              -D heron.kubernetes.scheduler.uri=http://localhost:8001
              -D heron.executor.docker.image=heron/heron:latest
              -D heron.class.uploader=org.apache.heron.uploader.dlog.DLUploader
              -D heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/distributedlog
        - name: kubectl-proxy
          image: heron/kubectl:latest
          command: ["sh", "-c"]
          args:
            - >
              kubectl proxy -p 8001
---    
apiVersion: v1
kind: Service
metadata:
  name: heron-apiserver
spec:
  selector:
    app: heron-apiserver
  ports:
    - protocol: TCP
      port: 9000
      targetPort: 9000
  type: NodePort

and this is my response when I curl the api server:

{
  "heron.build.git.revision" : "ddbb98bbf173fb082c6fd575caaa35205abe34df",
  "heron.build.git.status" : "Clean",
  "heron.build.host" : "ci-server-01",
  "heron.build.time" : "Sat Mar 31 09:27:19 UTC 2018",
  "heron.build.timestamp" : "1522488439000",
  "heron.build.user" : "release-agent",
  "heron.build.version" : "0.17.8"
}

[sebastienpattyn93]

@cckellogg any updates on this?

[cristobalcl]

Last week I posted an issue very similar to this one: #2919

As this issue included an error message in the CLI side like this Failed to instantiate uploader class 'org.apache.heron.uploader.dlog.DLUploader' and I don't, I thought at that moment they should be different problems. But now I'm thinking they are related and maybe we are using different CLI versions, because the errors in the server side are similar. I posted there a full server log.

Any progress on this? Did you get over somehow?

[sprasad09]

Getting the same error :(

[2018-08-20 22:55:33 -0700] [INFO]: Launching topology: 'WindowedWordCount' [2018-08-20 22:55:33 -0700] [INFO]: {'config_property': [], 'topology-file-name': '/Users/sprasad/.heron/examples/heron-streamlet-examples.jar', 'verbose': False, 'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'sprasad', 'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 'role': 'sprasad', 'dry_run_format': 'colored_table', 'dry_run': False, 'config_path': '/usr/local/heron/conf', 'topology_main_jvm_property': [], 'environ': 'default', 'service_url': 'http://localhost:8001/api/v1/proxy/namespaces/default/services/heron-apiserver:9000', 'topology-class-name': 'com.twitter.heron.examples.streamlet.WindowedWordCountTopology'} [2018-08-20 22:55:33 -0700] [ERROR]: Failed to instantiate uploader class 'org.apache.heron.uploader.dlog.DLUploader' [2018-08-20 22:55:33 -0700] [ERROR]: Failed to launch topology 'WindowedWordCount'

[xiaoyao1991]

I had similar issues trying to run it on minikube locally. Turns out that the docker image on docker hub is outdated. The artifact in the docker image have com.twitter.... as namespaces instead of org.apache.....

When following the instructions on https://apache.github.io/incubator-heron/docs/operators/deployment/schedulers/kubernetes/#minikube:

1. Before running $ kubectl create -f https://raw.githubusercontent.com/twitter/heron/master/deploy/kubernetes/minikube/apiserver.yaml
2. Locally modify heron/deploy/kubernetes/minikube/apiserver.yaml
3. Change -D heron.class.uploader=org.apache.heron.uploader.dlog.DLUploader to -D heron.class.uploader=com.twitter.heron.uploader.dlog.DLUploader.
4. Run $ kubectl create -f <path_to_your_source>/heron/deploy/kubernetes/minikube/apiserver.yaml.
5. Then follow through the rest of the instructions should work.

[xiaoyao1991]

I also happened to find that the service_url on the doc does not work for my minikube setting on mac. I ended up using http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver/proxy as my service_url

[zhangcheng]

Indeed this help, thanks @xiaoyao1991 for the tips