Programmatic access to Kubeflow applications

[surajkota]

Is your feature request related to a problem? Please describe. How can a Kubeflow profile user programmatically create and mange resources in Kubeflow (from outside the cluster). e.g. create and track pipeline runs, create inference service, training jobs etc.

Provide guidance and best practices

Describe the solution you'd like Starting Reference: https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/

Describe alternatives you've considered TBD

[surajkota]

Programmatic access to Kubeflow Pipelines using an in-cluster pod

Kubeflow pipeline uses Argo Workflows in the backend. It has its own API server and only tracks the workflows created by the pipelines service, in short, it is not a CRD and controller model like some of the other components like Notebooks, Tensorboard, KServe etc. Hence, a user needs to authenticate to the pipeline service in order to run pipelines.

To access pipelines service from outside the cluster, one can obtain the cookies and pass it in the kfp SDK but it may not be suitable for system accounts. Further, for users using Cognito based deployment, currently it is not possible to programatically authenticate a request that uses Amazon Cognito for user authentication through Load Balancer, i.e. you cannot generate AWSELBAuthSessionCookie cookies by using the access tokens from Cognito. Hence we recommend the following way to programmatically access pipelines from outside the cluster:

Authenticate to your cluster by using an IAM role or using an ODIC provider and create a Job/pod which mounts the service token issued by the pipelines service. Use this job as a proxy to run pipelines. The following sample list_experiements in kubeflow-user-example-com profile:

Note:

• Make sure you change the kubeflow-user-example-com in spec.containers.command to your namespace
• kfp sdk is already installed in the image used by the pod below.
• Profile controller sets up the permission for the default-editor service account to access pipelines.

Export the profile namespace

export PROFILE_NAMESPACE=kubeflow-user-example-com

Create the pod

cat <<EOF > access_pipelines.yaml
apiVersion: v1
kind: Pod
metadata:
  name: access-kfp-example
  namespace: $PROFILE_NAMESPACE
spec:
  serviceAccountName: default-editor
  containers:
  - image: public.ecr.aws/kubeflow-on-aws/notebook-servers/jupyter-pytorch:1.12.1-cpu-py38-ubuntu20.04-ec2-v1.2
    name: connect-to-pipeline
    command: ['python', '-c', 'import kfp; namespace = "$PROFILE_NAMESPACE"; client = kfp.Client(); print(client.list_experiments(namespace=namespace))']
    env:
      - ## this environment variable is automatically read by 
        ## this is the default value, but we show it here for clarity
        name: KF_PIPELINES_SA_TOKEN_PATH
        value: /var/run/secrets/kubeflow/pipelines/token
    volumeMounts:
      - mountPath: /var/run/secrets/kubeflow/pipelines
        name: volume-kf-pipeline-token
        readOnly: true
  volumes:
    - name: volume-kf-pipeline-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
              audience: pipelines.kubeflow.org
EOF
kubectl apply -f access_pipelines.yaml

To monitor the run,

kubectl logs -n $PROFILE_NAMESPACE access-kfp-example

[surajkota]

Access Kubeflow Pipelines using cookies with Dex as auth provider

Following is an end to end sample of triggering Kubeflow Pipelines from outside the cluster based on https://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/#example-for-dex which runs a toy pipeline to adds 2 numbers

Connect to Kubeflow endpoint

• Option1: By setting up an external endpoint using AWS ALB. Follow - https://awslabs.github.io/kubeflow-manifests/docs/add-ons/load-balancer/guide/
• Option2: By port forwarding to ingressgateway. Run in a separate terminal kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Run a Pipeline

• Specify the values for KUBEFLOW_ENDPOINT, PROFILE_USERNAME, PROFILE_PASSWORD and PROFILE_NAMESPACE for running the following python script to connect to kubeflow, get auth session cookie and run a pipeline

# e.g. "https://kubeflow.platform.example.com" if you followed load balancer guide
# or "http://localhost:8080" if using port-forward  
KUBEFLOW_ENDPOINT=""
# e.g. [email protected]
PROFILE_USERNAME=""
# password for the user
PROFILE_PASSWORD=""
# namespace for the profile
PROFILE_NAMESPACE="kubeflow-user-example-com"

import kfp
import requests
from kfp.components import create_component_from_func
from datetime import datetime

def get_auth_session_cookie(host, login, password):
    session = requests.Session()
    response = session.get(host)
    headers = {
        "Content-Type": "application/x-www-form-urlencoded",
    }
    data = {"login": login, "password": password}
    session.post(response.url, headers=headers, data=data)
    session_cookie = session.cookies.get_dict()["authservice_session"]
    return session_cookie

session_cookie = get_auth_session_cookie(
        KUBEFLOW_ENDPOINT, PROFILE_USERNAME, PROFILE_PASSWORD
)
print(f"retrieved cookie: {session_cookie}")

# Connect to KFP and create an experiment
kfp_client = kfp.Client(host=f"{KUBEFLOW_ENDPOINT}/pipeline", cookies=f"authservice_session={session_cookie}", namespace=PROFILE_NAMESPACE)
exp_name = datetime.now().strftime("%Y-%m-%d-%H-%M")
experiment = kfp_client.create_experiment(name=f"demo-{exp_name}")

# Run a sample pipeline
# https://www.kubeflow.org/docs/components/pipelines/v1/sdk/python-function-components/#getting-started-with-python-function-based-components

def add(a: float, b: float) -> float:
  '''Calculates sum of two arguments'''
  return a + b

add_op = create_component_from_func(
    add, output_component_file='add_component.yaml')

import kfp.dsl as dsl
@dsl.pipeline(
  name='Addition pipeline',
  description='An example pipeline that performs addition calculations.'
)
def add_pipeline(
  a='1',
  b='7',
):
  # Passes a pipeline parameter and a constant value to the `add_op` factory
  # function.
  first_add_task = add_op(a, 4)
  # Passes an output reference from `first_add_task` and a pipeline parameter
  # to the `add_op` factory function. For operations with a single return
  # value, the output reference can be accessed as `task.output` or
  # `task.outputs['output_name']`.
  second_add_task = add_op(first_add_task.output, b)

# Specify argument values for your pipeline run.
arguments = {'a': '7', 'b': '8'}

# Create a pipeline run, using the client you initialized in a prior step.
run = kfp_client.create_run_from_pipeline_func(add_pipeline, arguments=arguments, experiment_name=experiment.name)
print(run.run_info)

[tjhorner]

Hi, just wanted to chime in here. I was tinkering with a way to enable programmatic access to the KFP API today, and I think it's a pretty decent solution. Since Istio supports JWT authentication, we decided to use that plus a new AuthorizationPolicy to allow JWT-auth'd access into the ml-pipeline-ui (tried using ml-pipeline at first, but there is a conflicting AuthorizationPolicy that's too permissive). And since the existing ALB is configured to always use Cognito auth, we had to create a new one which points directly to ml-pipeline-ui and delegates authentication to Istio.

Once that's set up, you can obtain a JWT from the issuer using a M2M flow (in our case, we use Auth0) and provide that in the Authorization header with requests to the new ALB.

Here are example manifests for the resources we created:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: ml-pipeline-ui-jwt
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: ml-pipeline-ui
  jwtRules:
  - issuer: "https://example.com"
    jwksUri: "https://example.com/.well-known/jwks.json"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ml-pipeline-ui-require-jwt
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: ml-pipeline-ui
  action: ALLOW
  rules:
  - from:
    - source:
       requestPrincipals: ["https://example.com/[email protected]"]

The above allows JWTs issued by https://example.com to subject [email protected] to access the service.

Then, the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: ARN_FOR_CERT # replace me
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=false
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
  name: kfp-ingress
  namespace: kubeflow
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: ml-pipeline-ui
            port:
              number: 80
        path: /*
        pathType: ImplementationSpecific

Then you can make authenticated requests to it, for example to retrieve all pipelines:

$ curl -H "Authorization: Bearer $JWT" -H "kubeflow-userid: $KF_USER" https://$ALB_NAME/apis/v1beta1/pipelines

There are some improvements to be made here, for example copying the sub claim to the kubeflow-userid header instead of setting it manually, but it's better than the alternatives (i.e., performing the whole Cognito UI auth flow in code then retrieving the cookie) and works well for our use case.

[ranjanshivaji]

We have the kubeflow deployment on AWS. With S3, RDS and Cognito. What way do you suggest to programmatically access the pipelines and run them. We are using the 1.6v

[gioargyr]

I have Kubeflow on premises. I switched from Dex to Keycloak. Does anyone have any clue how to access programmatically Kubeflow pipelines from outside the cluster?

[thesuperzapper]

@gioargyr since most Kubeflow deployments use Dex, you will probably have to make a custom solution.

For those who want a dex-based solution, the deployKF docs give some good examples of how to authenticate with Kubeflow Pipelines.

Most of these will work with any Dex-based Kubeflow, but the "Browser Login Flow" requires that you use deployKF. Its sort of like how AWS CLI authenticates, where the user is given a browser link to open and authenticate.

[gioargyr]

I found it by using Python kfp ! After a lot of testing, I realized -for one more time- that Python kfp has bad documentation: https://kubeflow-pipelines.readthedocs.io/en/stable/source/client.html

From this page https://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/ we learn that:

• If you are inside the cluster, you need to make you environment have KF_PIPELINES_SA_TOKEN_PATH env correctly defined (to be specific, this env should point to a file that holds the Service Account token of the Kubernetes Service Account who is acting in your environment). You then "force" kfp.Client to use this env for authentication by defining credentials like this credentials = kfp.auth.ServiceAccountTokenVolumeCredentials(path=None) The Kubeflow client is now defined like this: client = kfp.Client(host=<KF-pipelines-url>, credentials=credentials)

• If you are outside the cluster, you need to authenticate through Dex. So, first you need to get the cookies or session_cookie from Dex. Then you define kfp client like this: client = kfp.Client(host=<KF-pipelines-url>, cookies=<cookie-from-Dex>)

However, as we can see in kfp.Client documentation https://kubeflow-pipelines.readthedocs.io/en/stable/source/client.html we would expect that there are several arguments to use with "intuitive" names like client_id, other_client_id and existing_token. Spoiler alert: None of them worked for me (and as I said, they are badly documented).

• So, if you use Keycloak instead of Dex:

First you need to authenticate to Keycloak and want to programmatic access to Kubeflow pipelines outside the cluster:

First you need to authenticate to Keycloak:

from keycloak import KeycloakOpenID

keycloak_url = "<KEYCLOAK-URL>/realms"
realm_name = "<REALM_NAME>"
client_id = "<KUBEFLOW-ID-AS-KEYCLOAK-CLIENT>"
client_secret = "<KUBEFLOW-SECRET-AS-KEYCLOAK-CLIENT>"

kc_openid = KeycloakOpenID(server_url=keycloak_url, client_id=client_id, realm_name=realm_name, client_secret_key=client_secret)

username = "<USERNAME_KEYCLOAK-USER>"
password = "<PASSWORD_KEYCLOAK-USER>"

token = kc_openid.token(username, password)

This token is a dictionary and 3 of its contents are access_token, refresh_token and id_token. What worked for me was to act like I was inside cluster. I store the id_token in a file (e.g. /token) I force kfp client to use the KF_PIPELINES_SA_TOKEN_PATH env like I am inside the cluster I define the env to point to the file where the id_token is: KF_PIPELINES_SA_TOKEN_PATH=/token

Which means that the Keycloak id_token is the correct "replacement" for the Service Account token! Neither intuitive, nor documented anywhere! (Correct me if I am wrong. I am very curious!)

[afrozsh19]

Thanks @gioargyr - I tried your solution, and it worked perfectly for me. There isn't clear documentation on the Kubeflow Pipelines API Client.

Environment details:

• Kubeflow Pipeline SDK - v2.5.0
• Authentication through Keycloak using OIDC
• Kubeflow Pipelines - v2.0.5

Below is the code snippet I used, which others can also follow, keycloak authentication code is referenced from https://github.com/awslabs/kubeflow-manifests/issues/493#issuecomment-2145062847:

from keycloak import KeycloakOpenID
import os
import kfp
from kfp.client.set_volume_credentials import ServiceAccountTokenVolumeCredentials

# for kubeflow pipelines v1
# from kfp.auth import ServiceAccountTokenVolumeCredentials

# Keycloak configuration
keycloak_url = "<KEYCLOAK-URL>/realms"
client_id = "<KUBEFLOW-ID-AS-KEYCLOAK-CLIENT>"
client_secret = "<KUBEFLOW-SECRET-AS-KEYCLOAK-CLIENT>"

# User credentials
username = "<USERNAME_KEYCLOAK-USER>"
password = "<PASSWORD_KEYCLOAK-USER>"

# Initialize Keycloak client
kc_openid = KeycloakOpenID(server_url=keycloak_url,
                           client_id=client_id,
                           realm_name="<REALM_NAME>",
                           client_secret_key=client_secret)

# Authenticate and get the token
token = kc_openid.token(username, password)

# Extract the id_token
id_token = token['id_token']

# Save the id_token to a file (e.g., /tmp/token)
token_path = "/tmp/token"
with open(token_path, 'w') as token_file:
    token_file.write(id_token)

# Set the KF_PIPELINES_SA_TOKEN_PATH environment variable
os.environ['KF_PIPELINES_SA_TOKEN_PATH'] = token_path

# Initialize the KFP client
credentials = ServiceAccountTokenVolumeCredentials(path=None)
client = kfp.Client(host="<KF_PIPELINES_URL>", credentials=credentials)

print("Kubeflow Pipelines client initialized.")

Is This Solution a Best Practice?

The solution is effective, however, there are a few considerations:

• Storing tokens in files can be a security risk if not managed properly. Ensure that the file storing the id_token has appropriate permissions and is not accessible by unauthorized users.
• Keycloak tokens typically have an expiration time. You may need to handle token refreshes.