tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

engine job can't be stopped

Open lance6716 opened this issue 3 years ago • 4 comments

What did you do?

$ curl -X POST http://127.0.0.1:10245/api/v1/jobs/a2728c04-c963-4505-894f-b120d417fff7/cancel
{"id":"a2728c04-c963-4505-894f-b120d417fff7", "type":"DM", "status":"Canceling", "config":"bmFtZTogdGVzdAp0YXNrLW1vZGU6IGFsbApjb2xsYXRpb25fY29tcGF0aWJsZTogInN0cmljdCIKdGFyZ2V0LWRhdGFiYXNlOgogIGhvc3Q6IGhvc3QuZG9ja2VyLmludGVybmFsCiAgcG9ydDogNDAwMAogIHVzZXI6IHJvb3QKICBwYXNzd29yZDogJycKdXBzdHJlYW1zOgogIC0gZGItY29uZmlnOgogICAgICBob3N0OiBob3N0LmRvY2tlci5pbnRlcm5hbAogICAgICBwb3J0OiAzMzA2CiAgICAgIHVzZXI6IHJvb3QKICAgICAgcGFzc3dvcmQ6ICcnCiAgICBzb3VyY2UtaWQ6IG15c3FsLTAxCiAgICBibG9jay1hbGxvdy1saXN0OiBnbG9iYWwKICAtIGRiLWNvbmZpZzoKICAgICAgaG9zdDogaG9zdC5kb2NrZXIuaW50ZXJuYWwKICAgICAgcG9ydDogMzMwNwogICAgICB1c2VyOiByb290CiAgICAgIHBhc3N3b3JkOiAnJwogICAgc291cmNlLWlkOiBteXNxbC0wMgogICAgYmxvY2stYWxsb3ctbGlzdDogZ2xvYmFsCmJsb2NrLWFsbG93LWxpc3Q6CiAgZ2xvYmFsOgogICAgZG8tZGJzOgogICAgLSAic3luY19jb2xsYXRpb24qIgogICAgLSAidGVzdF9wYW5pYyIK"}

$ curl -X DELETE http://127.0.0.1:10245/api/v1/jobs/a2728c04-c963-4505-894f-b120d417fff7
{"code":9, "details":[{"@type":"type.googleapis.com/enginepb.ErrorV2", "name":"JobNotStoppedError", "details":"eyJKb2JJRCI6ImEyNzI4YzA0LWM5NjMtNDUwNS04OTRmLWIxMjBkNDE3ZmZmNyJ9", "stack_trace":"\ngithub.com/pingcap/tiflow/engine/pkg/rpcerror.(*Prototype[...]).GenWithStack\n\tgithub.com/pingcap/tiflow/engine/pkg/rpcerror/normalize.go:101\ngithub.com/pingcap/tiflow/engine/servermaster.(*JobManagerImpl).DeleteJob\n\tgithub.com/pingcap/tiflow/engine/servermaster/jobmanager.go:136\ngithub.com/pingcap/tiflow/engine/servermaster.(*Server).DeleteJob\n\tgithub.com/pingcap/tiflow/engine/servermaster/server.go:297\ngithub.com/pingcap/tiflow/engine/enginepb._JobManager_DeleteJob_Handler.func1\n\tgithub.com/pingcap/tiflow/engine/enginepb/master_grpc.pb.go:605\ngithub.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/[email protected]/server_metrics.go:107\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\tgoogle.golang.org/[email protected]/server.go:1129\ngithub.com/pingcap/tiflow/engine/pkg/rpcerror.UnaryServerInterceptor\n\tgithub.com/pingcap/tiflow/engine/pkg/rpcerror/grpc.go:140\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\tgoogle.golang.org/[email protected]/server.go:1132\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\tgoogle.golang.org/[email protected]/server.go:1134\ngithub.com/pingcap/tiflow/engine/enginepb._JobManager_DeleteJob_Handler\n\tgithub.com/pingcap/tiflow/engine/enginepb/master_grpc.pb.go:607\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/[email protected]/server.go:1295\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/[email protected]/server.go:1636\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/[email protected]/server.go:932\nruntime.goexit\n\truntime/asm_amd64.s:1594"}]}

$ curl http://127.0.0.1:10245/api/v1/jobs/a2728c04-c963-4505-894f-b120d417fff7
{"id":"a2728c04-c963-4505-894f-b120d417fff7", "type":"DM", "status":"Running", "config":"bmFtZTogdGVzdAp0YXNrLW1vZGU6IGFsbApjb2xsYXRpb25fY29tcGF0aWJsZTogInN0cmljdCIKdGFyZ2V0LWRhdGFiYXNlOgogIGhvc3Q6IGhvc3QuZG9ja2VyLmludGVybmFsCiAgcG9ydDogNDAwMAogIHVzZXI6IHJvb3QKICBwYXNzd29yZDogJycKdXBzdHJlYW1zOgogIC0gZGItY29uZmlnOgogICAgICBob3N0OiBob3N0LmRvY2tlci5pbnRlcm5hbAogICAgICBwb3J0OiAzMzA2CiAgICAgIHVzZXI6IHJvb3QKICAgICAgcGFzc3dvcmQ6ICcnCiAgICBzb3VyY2UtaWQ6IG15c3FsLTAxCiAgICBibG9jay1hbGxvdy1saXN0OiBnbG9iYWwKICAtIGRiLWNvbmZpZzoKICAgICAgaG9zdDogaG9zdC5kb2NrZXIuaW50ZXJuYWwKICAgICAgcG9ydDogMzMwNwogICAgICB1c2VyOiByb290CiAgICAgIHBhc3N3b3JkOiAnJwogICAgc291cmNlLWlkOiBteXNxbC0wMgogICAgYmxvY2stYWxsb3ctbGlzdDogZ2xvYmFsCmJsb2NrLWFsbG93LWxpc3Q6CiAgZ2xvYmFsOgogICAgZG8tZGJzOgogICAgLSAic3luY19jb2xsYXRpb24qIgogICAgLSAidGVzdF9wYW5pYyIK"}

What did you expect to see?

No response

What did you see instead?

job can be stopped

Versions of the cluster

master

lance6716 avatar Aug 23 '22 07:08 lance6716

df.tar.gz

lance6716 avatar Aug 23 '22 07:08 lance6716

caused by https://github.com/pingcap/tiflow/issues/6749

lance6716 avatar Aug 23 '22 07:08 lance6716

@amyangfei Can we close this issue now?

maxshuang avatar Sep 09 '22 09:09 maxshuang

I will check it later

amyangfei avatar Sep 09 '22 09:09 amyangfei