Request unauthorized for admin search-attributes commands

[TheGeniesis]

Expected Behavior

I can execute CRUD commands with admin x-search-attributes without errors.

Actual Behavior

I have devl:admin and temporal-system:admin roles assigned. I use only sql, postgres12 driver for "default" and "visibility" dbs.

When I execute one of x-search-attributes command inside admintools pod it fails with error:

I use devl namespace as an example of not working scenario

tctl --namespace=devl admin cluster get-search-attributes
Error: Unable to get search attributes.
Error Details: rpc error: code = PermissionDenied desc = Request unauthorized.

When I run this command with not existing namespace I get different error:

Error Details: rpc error: code = NotFound desc = Namespace searchtest2 is not found.

Cluster is working:

tctl cluster health
temporal.api.workflowservice.v1.WorkflowService: SERVING

I tested other admin commands (provided auth token as an env variable) related to cluster info (tctl --namespace=devl admin cluster describe, tctl --namespace=devl admin cluster list ), namespaces (tctl --namespace=devl namespace list, tctl --namespace=devl namespace describe), etc. and they are working. Only search-attributes has this problem. Non-admin command (tctl --namespace=devl cluster get-search-attributes) works as expected. Unfortunately only admin command allows to add attributes.

Log from frontend pod:

 {"level":"error","ts":"2024-10-15T13:59:29.365Z","msg":"service failures","operation":"AdminGetSearchAttributes","wf-namespace":"devl","error":"Unable to get namespace devl info with error: Request unauthorized.","logging-call-at":"telemetry.go:411","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:411\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:187\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:181\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:85\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:135\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/utf8validator.(*Validator).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/utf8validator/validate.go:182\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1177\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/adminservice/v1/service_grpc.pb.go:1062\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1019"}

Stacktrace from command:

Error: Unable to get search attributes.
Error Details: rpc error: code = Unavailable desc = Unable to get namespace devl info with error: Request unauthorized.
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:16 +0x13
github.com/temporalio/tctl/cli_curr.printError({0x1990e48, 0x20}, {0x1dd3940, 0xc00051c000})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:393 +0x218
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0x1990e48?, 0x1dfaf50?}, {0x1dd3940?, 0xc00051c000?})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:404 +0x25
github.com/temporalio/tctl/cli_curr.AdminGetSearchAttributes(0xc0005371e0)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin_cluster_search_attributes_commands.go:157 +0x89
github.com/temporalio/tctl/cli_curr.newAdminClusterCommands.func3(0xc0005371e0?)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin.go:496 +0x13
github.com/urfave/cli.HandleAction({0x16a4dc0?, 0x1a3db98?}, 0x15?)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:526 +0x75
github.com/urfave/cli.Command.Run({{0x1978c96, 0x15}, {0x0, 0x0}, {0xc0006cb080, 0x1, 0x1}, {0x198e717, 0x1f}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:173 +0x63e
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000239880, 0xc000536f20)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x7e5
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0002396c0, 0xc000536dc0)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x7e5
github.com/urfave/cli.(*App).Run(0xc000239340, {0xc00003e0a0, 0x5, 0x5})
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:277 +0xb27
main.main()
	/home/runner/work/docker-builds/docker-builds/tctl/cmd/tctl/main.go:47 +0xa5

Steps to Reproduce the Problem

1. Deploy temporal on AKS cluster using modified chart
2. Create proper groups, users and service principals (I followed medium article with small tweaks, since it wasn't 100% up-to-date)
3. Login into admintools pod
4. Create a new namespace using UI session temporal operator namespace create --namespace <namespace> --grpc-meta=Authorization='Bearer <token_from_ui>' (works)
5. Run any of admin x-search-attributes command tctl --namespace=devl cluster get-search-attributes --auth='Bearer <token_from_ui>' or temporal operator search-attribute create --name email --type Keyword --grpc-meta=Authorization='Bearer <token_from_ui>'

Specifications

• Version: 1.24.2 - after upgrade to 1.25.1 problem still exists
• Chart version: 0.44.0 (modified to enable internal frontend and oAuth) - after upgrade to 0.50.0 problem still exists

Question

Do I miss something related to role assignment? I searched frontend, admintools and worker logs, but I couldn't find anything which might help me to debug this problem.

[TheGeniesis]

I did one more test - I enabled ES (changed flag elasticsearch.enabled: true in a chart and the feature works.

I checked Temporal documentation and I shouldn't have any problems with using search attributes without ES.

If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the supported SQL databases with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.

I just wonder if this is the reason of the problem. For PSQL it should be namespace specific, but command was created for admin.

I also checked UI and https://<host>/api/v1/namespaces/devl/search-attributes? returns 503 in the UI when PSQL is enabled (no problem with ES).

[TheGeniesis]

For 503 error I added additional logs into the code and found that claims are not send for DescribeNamespace:

Changes in default_authorizer.go

func (a *defaultAuthorizer) Authorize(_ context.Context, claims *Claims, target *CallTarget) (Result, error) {
	// APIs that are essentially read-only health checks with no sensitive information are
	// always allowed
	if IsHealthCheckAPI(target.APIName) {
		return resultAllow, nil
	}
	fmt.Println("Check claims for ", target.APIName)
	dbg2(claims)
	if claims == nil {
		return resultDeny, nil
	}
	metadata := api.GetMethodMetadata(target.APIName)
	fmt.Println("API nAME", target.APIName)

Logs from frontend service:

Check claims for  /temporal.api.operatorservice.v1.OperatorService/ListSearchAttributes
{"Subject":"DmXTqIs843B3i6f-AnAcr2cakEpztzAmMrCuG-d8DXM","System":9,"Namespaces":{"devl":9,"intg":9,"uacc":9},"Extensions":null}
API nAME /temporal.api.operatorservice.v1.OperatorService/ListSearchAttributes
hasRole 9

Check claims for  /temporal.api.workflowservice.v1.WorkflowService/DescribeNamespace
null

In docker-builds/temporal/service/frontend/admin_handler.go I see function getSearchAttributesSQL which executes the DescribeNamespace function.

Is there any easy way to add claims to the function execution?

Edit:

To confirm I added a condition to bypass claims check for DescribeNamespace endpoint and UI started working.

if (strings.Contains(target.APIName, "DescribeNamespace")) {
  return resultAllow, nil
}

Edit 2: After upgrade to the newest version I see commands to add search-attributes on operator level. I added the same bypass for UpdateNamespace and executed command - it works.

[bergundy]

Thanks for the report. While these admin APIs are deprecated and so is tctl there may still be an actual issue here and we'll look into it.

[underrun]

This is still an issue - also it's got nothing to do with deprecated admin apis or tctl.

The place i noticed this is in the web ui when i hit a namespace endpoint at a path like /namespaces/hello-world

The behavior is that the top half of the page renders but i get nothing under search attributes and a little 503 toast pops up at the bottom - which happens because ListSearchAttributesSQL performs an additional client call to frontend itself to DescribeNamespace but this client call can only use the level of auth allowed by the frontend's own claims.

This can for sure be worked around by allowing whatever auth's frontend to frontend to be allowed to describe a namespace even if that identity hasn't been granted specific access to that namespace, but this shouldn't really be necessary.

there's an obvious amount of access that the server has to have to do stuff to whatever is running in the server, but it would still be nice to be able to only allow those actions that are necessary.

the workaround i've taken is to make namespaced api calls for identities whose claims don't have that namespace as part of the claim only be authorized if their system roles include both Admin and whatever the other role that's required (for describe it'd be Read).

but the issue with that is that it also means any users with system role of admin can do more than what I would really like for them to do. i could maybe special case things that are identified as temporal server specifically but i'd rather keep my auth based on roles rather than specific identities for reasons...

so this isn't a "big deal" but it make my authorizer logic more complex than it needs to be and makes it a little difficult to separate users who are admins and temporal server service workloads from each other.