Unable to call matching.PollActivityTaskQueue

[Baskkra]

Expected Behavior

temporal can running as it was

Actual Behavior

temporal can't be called by service that used it

Steps to Reproduce the Problem

1. temporal up and running
2. our service try to create temporal client init
3. and got this error : temporal front-end {"level":"error","ts":"2022-08-08T05:30:42.376Z","msg":"Unable to call matching.PollActivityTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/CMNINV-ADJUST_STOCK_BY_DELTA-SET_CHANNEL_STOCK/2","timeout":"1m9.999634523s","error":"any: message type \"temporal.api.errordetails.v1.NamespaceNotFoundFailure\" isn't linked in","logging-call-at":"workflowHandler.go:1047","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:1047\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:502\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:498\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/[email protected]/workflowservice/v1/service.pb.go:1142\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:90\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1127\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryServerInterceptors.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1113\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/[email protected]/workflowservice/v1/service.pb.go:1144\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934"}

temporal-web : metadata: Metadata { _internal_repr: {}, flags: 0 }, details: 'service rate limit exceeded' } [2022-08-08T05:28:11.082Z] OperationalError: 8 RESOURCE_EXHAUSTED: service rate limit exceeded at Object.exports.createStatusError (/usr/app/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/usr/app/node_modules/grpc/src/client_interceptors.js:1209:28) at InterceptingListener._callNext (/usr/app/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/usr/app/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/usr/app/node_modules/grpc/src/client_interceptors.js:847:24) { cause: Error: 8 RESOURCE_EXHAUSTED: service rate limit exceeded at Object.exports.createStatusError (/usr/app/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/usr/app/node_modules/grpc/src/client_interceptors.js:1209:28) at InterceptingListener._callNext (/usr/app/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/usr/app/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/usr/app/node_modules/grpc/src/client_interceptors.js:847:24) { code: 8, metadata: Metadata { _internal_repr: {}, flags: 0 }, details: 'service rate limit exceeded' }, isOperational: true, code: 8, metadata: Metadata { _internal_repr: {}, flags: 0 }, details: 'service rate limit exceeded' }

Specifications

• Version: temporal 1.11
• Platform: helm chart k8s

[yiminc]

Is your server having different version of frontend and matching? It looks like frontend is running an older version that does not recogonize NamespaceNotFoundFailure. Also you need to register your namespace first. You can do so using tctl tool.

[Baskkra]

Hi @yiminc thank you it happens because we have 2 different release in one k8s namespace that have same label with different version. It's fixed with moving other deployment to another namespace.

Super thank you for the response!