failed to authenticate with kubeflow pipeline [bug] <HTTP 500 error>

[LeBoyOrion]

• How do you deploy Kubeflow Pipelines (KFP)?

Kubeflow is deployed on MiniKF locally downloaded on windows using vagrant and Virtual Box downloading steps : here

• KFP version: i cant fin it, neither on the dahsboard nor on the minikf window.

Steps to reproduce

The problem can be reproduced just by using Kale to create a pipeline and after dividing any notebook into different steps, clicking on compile and run shows an error that i can find by going to the pipeline window logs, the logs show me this :

the second error message is really long, if someone needs it to maybe understand my issue you will find it as a comment to this post

Expected result

After seeing a lot of videos that implements kale and uses it to create pipelines, the expected result is just a pipeline that runs every cells of the notebook contained in each step using the data of the step before.

Thank you in advance for your help.

Impacted by this bug? Give it a 👍.

[LeBoyOrion]

the second error message :

HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).canAccessRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:390\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:151\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize the request\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:153\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","code":13,"message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).canAccessRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:390\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:151\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize the request\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:153\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).canAccessRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:390\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:151\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize the request\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:153\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","error_details":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).canAccessRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:390\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:151\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize the request\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRun\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:153\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1519\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._RunService_GetRun_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/run.pb.go:1521\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"}]}

[connor-mccarthy]

/cc @zijianjoy

[zijianjoy]

Question: Which endpoint are you trying to reach within this load step? It looks like you need to authenticate before making request: Some documentation might be relevant in terms of making network call within cluster: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/#multi-user-mode

[kdubovikov]

@LeBoyOrion If you are trying to access Kubeflow Pipelines SDK from a Notebook instance inside Kubeflow, then I think I know what the problem is.

You can create this PodDefault definition:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: access-ml-pipeline
  namespace: kubeflow-user-example-com
spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
    matchLabels:
      access-ml-pipeline: "true"
  volumes:
    - name: volume-kf-pipeline-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
              audience: pipelines.kubeflow.org
  volumeMounts:
    - mountPath: /var/run/secrets/kubeflow/pipelines
      name: volume-kf-pipeline-token
      readOnly: true
  env:
    - name: KF_PIPELINES_SA_TOKEN_PATH
      value: /var/run/secrets/kubeflow/pipelines/token

Then, select "Allow access to Kubeflow Pipelines" configuration when creatine a Notebook in Kubeflow. Afterwards, you can create Kubeflow Pipelines client like this:

with open(os.environ['KF_PIPELINES_SA_TOKEN_PATH'], "r") as f:
    TOKEN = f.read()

client = kfp.Client(
    host='http://ml-pipeline.kubeflow.svc.cluster.local:8888',
    existing_token=TOKEN,

[tam0201]

Hi, i am trying to build my own api on top of the sdk client. My kubeflow is full deployment on GCP. I am attempted to call client.list_experiments(), but it shows the same error as LeBoyOrion. Any suggestion?

Note: I am using FastAPI and built an website in the same cluster. I am able to call the list_pipelines API but failed to list runs or experiments.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rimolive]

Closing this issue. No activity for more than a year.

/close

[google-oss-prow[bot]]

@rimolive: Closing this issue.

In response to this:

Closing this issue. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.