flinkk8soperator icon indicating copy to clipboard operation
flinkk8soperator copied to clipboard

TaskExecutor could not resolve address.

Open mikechengwei opened this issue 5 years ago • 1 comments

Error Log

2019-09-25 07:45:37,610 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:45:57,639 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:17,680 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:37,719 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:46:57,759 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:17,779 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:37,819 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..
2019-09-25 07:47:57,849 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-68077756:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@wordcount-operator-example-68077756:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Sender[null] sent message of type "akka.actor.Identify"..

deploy yaml

apiVersion: flink.k8s.io/v1beta1
kind: FlinkApplication
metadata:
  name: wordcount-operator-example
  namespace: flink-operator
  annotations:
  labels:
    environment: development
spec:
  image: lyft/wordcount-operator-example:500fe6bd40da8efca4a48bbb1104896be2c1fae8
  flinkConfig:
    taskmanager.heap.size: 200
    state.backend.fs.checkpointdir: file:///checkpoints/flink/checkpoints
    state.checkpoints.dir: file:///checkpoints/flink/externalized-checkpoints
    state.savepoints.dir: file:///checkpoints/flink/savepoints
  jobManagerConfig:
    resources:
      requests:
        memory: "200Mi"
        cpu: "0.2"
    replicas: 1
  taskManagerConfig:
    taskSlots: 2
    resources:
      requests:
        memory: "200Mi"
        cpu: "0.2"
  flinkVersion: "1.8"
  jarName: "wordcount-operator-example-1.0.0-SNAPSHOT.jar"
  parallelism: 3
  entryClass: "org.apache.flink.WordCount"

mikechengwei avatar Sep 25 '19 08:09 mikechengwei

Was the job manager having issues ? Did you check logs in the Jobmanager ?

2019-09-25 19:49:29,483 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@wordcount-operator-example-8e8ca53d:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@wordcount-operator-example-8e8ca53d:6123/user/resourcemanager..
2019-09-25 19:49:29,517 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with org.apache.flink.shaded.akka.org.jboss.netty.channel.ConnectTimeoutException: connection timed out: wordcount-operator-example-8e8ca53d/10.108.150.41:6123
2019-09-25 19:49:39,650 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Resolved ResourceManager address, beginning registration
2019-09-25 19:49:39,650 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Registration at ResourceManager attempt 1 (timeout=100ms)
2019-09-25 19:49:39,726 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Successful registration at resource manager akka.tcp://flink@wordcount-operator-example-8e8ca53d:6123/user/resourcemanager under registration id 49f75b8fdafe26d466c412219741ed3f.

Also can you try increasing the cpu on the Jobmanager ?

anandswaminathan avatar Sep 28 '19 22:09 anandswaminathan