flinkk8soperator icon indicating copy to clipboard operation
flinkk8soperator copied to clipboard

FlinkOperator's state not working as expected

Open Rep1AI opened this issue 5 years ago • 1 comments

I submitted wordcount and other jobs..

I was expecting state flows as follow:

New --> ClusterStarting -> Running

But, When I see in UI, the job as finished already but in operator events it never showed ClusterStarting and Running... Then, Running to Updating Updating to ClusterStarting

From Running State, I was expecting it to go to Updating State. But all the logs show is SubmittingJob

On minimum from SubmittingJob to Running should have be captured right?

Logs:

kubectl logs -f -n flinkops flinkops-68b7cc8f7c-8pkzb

time="2020-02-23T18:38:31Z" level=info msg=------------------------------------------------------------------------
time="2020-02-23T18:38:31Z" level=info msg="Using config file: [/etc/flinkops/config/config.yaml]"
time="2020-02-23T18:38:31Z" level=info msg="Config section [logger] updated. Firing updated event."
{"json":{},"level":"info","msg":"Config section [operator] updated. No update handler registered.","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"\u0026{ResyncPeriod:30s LimitNamespace: MetricsPrefix:flinkk8soperator ProfilerPort:10254 FlinkIngressURLFormat: UseProxy:false ProxyPort:8001 ContainerNameFormat: Workers:4 BaseBackoffDuration:100ms MaxBackoffDuration:30s MaxErrDuration:5m0s}\n","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Registering Components.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Adding controllers.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Starting the Cmd.","ts":"2020-02-23T18:38:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Logged Normal event: CreatingCluster: Creating Flink cluster for deploy 7df2a560","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Flink cluster has started successfully","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"Savepointing"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"No job found for the application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Logged Normal event: JobSubmitted: Flink job submitted to cluster with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Conflict while updating status","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Failed to reconcile resource flink/wordcount-operator-example: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"wordcount-operator-example\": the object has been modified; please apply your changes to the latest version and try again","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Found already-submitted job for application with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:34Z"}



{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:04:04Z"}

Rep1AI avatar Feb 23 '20 19:02 Rep1AI

@RohitR1 This is just an edge case problem with the example I believe. The example we have is a batch job, and I think it finished. Can you check the status of the job?

anandswaminathan avatar Feb 25 '20 05:02 anandswaminathan