How to do programmatic authentication with Dex?

[jlewi]

@yanniszark @krishnadurai How do folks do programmatic authentication using DEX?

Here's the use case; suppose you are running code outside the cluster and you want to programmatically access a service running inside the cluster e.g.

• Talk to the KFP API Server to submit a pipeline
• Send an inference request to a model end point

My expectation would be that the client code can obtain a credential and add it as an Auth Header in order to pass the AuthZ check performed at the ISTIO ingress-gateway.

For reference we follow this pattern on GCP.

Here are the instructions for doing programmatic authentication with IAP https://cloud.google.com/iap/docs/authentication-howto

And for example here's the code in the KFP SDK for doing auth using IAP. https://github.com/kubeflow/pipelines/blob/9ad7d7dd9776ce75a83712f5723db2ef93ba5c26/sdk/python/kfp/_auth.py#L41

@animeshsingh @jinchihe We might want to think about building a Kubeflow Auth library as part of a Kubeflow SDK so that each application (KFP, KFServing, etc...) doesn't have to implement it themselves.

[krishnadurai]

@jlewi @yanniszark Here's an approach: Using Dex's Cross-client trust and authorized party to generate tokens for the SDK from oidc-authservice app.

1. User logs into oidc-authservice, and has a separate UI page to generate a token for the SDK client, possibly embedded in kubeflow.
2. If a user chooses to generate a token in oidc-authservice, create a new OAuth client for the SDK client through the oidc-authservice backend. This can be done by using dex's grpc client with 'oidc-authservice' added to its trusted peers.
3. Further, using this information about the SDK client, a new token for the SDK client can be issued with 'oidc-authservice' in its audience.

Haven't tested this approach yet.

[ryandawsonuk]

Yeah it's more complex than you'd expect as dex doesn't support the password grant flow. We (Seldon) have so far only been able to do it in flows that include the browser.

[ryandawsonuk]

FWIW, argocd uses dex and lets you login just from the CLI. But I believe it's doing that by actually running a browser in the background.

[ryandawsonuk]

The argocd CLI code to perform the login starts at https://github.com/argoproj/argo-cd/blob/master/cmd/argocd/commands/login.go

[maurogonzalez]

I was able to authenticate following this flow. Not sure if it's the correct path but it worked for me using this installation https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/#add-static-users-for-basic-auth.

0) curl -v http://$SERVICE:$PORT
Response:
>> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=STATE_VALUE">Found</a>.

STATE=STATE_VALUE

1) curl -v "http://$SERVICE:$PORT/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&amp;state=$STATE_VALUE"
Response:
>> <a href="/dex/auth/local?req=REQ_VALUE">Found</a>

REQ=REQ_VALUE
2) curl -v 'http://$SERVICE:$PORT/dex/auth/local?req=REQ_VALUE' -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'

3) curl -v 'http://$SERVICE:$PORT/dex/approval?req=$REQ_VALUE'

Response:
>> <a href="/login/oidc?code=CODE_VALUE&amp;state=STATE_VALUE">See Other</a>.

CODE=CODE_VALUE

4) curl -v 'http://$SERVICE:$PORT/login/oidc?code=$CODE_VALUE&amp;state=$STATE_VALUE'

Response:
>> set cookie authservice_session=SESSION

curl -v 'http://$SERVICE:$PORT/pipeline/apis/v1beta1/pipelines' -H 'Cookie: authservice_session=SESSION'

Response:
>> 200 OK { ... }

[Jeffwan]

@jlewi If IAP supports programmatic authentication, that means KFServing can use same istio ingress gateway. kfserving-ingressgateway without auth for IAP is not for authentication purpose but only knative probe issue?

[jlewi]

@Jeffwan I don't know the answer to that question; I had have to defer to the kfserving folks.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[ConverJens]

I'm having issues with programatically authenticating using dex for an on-prem install. Is there any progress on this or some standard way of authenticating using dex?

[rafbarr]

Running into this issue as well, which is unfortunately preventing us from upgrading to 1.1.

[ConverJens]

@rafaelbarreto87 For our purposes, this was solved. See: https://github.com/kubeflow/kubeflow/issues/5345

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/istio	0.56

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[rafbarr]

@rafaelbarreto87 For our purposes, this was solved. See: kubeflow/kubeflow#5345

@ConverJens Thanks a lot! This solves for some of our use cases as well (i.e. deploying pipelines from shell)! Really appreciated.

Unfortunately, I think it would be great to have a more permanent and general solution for this (e.g. OAuth2) because some processes are not interactive and this method depend on the UI, which is something that could easily change. Either way, it's a workable solution for now, so thanks a lot again!

[ConverJens]

I agree

@rafaelbarreto87 For our purposes, this was solved. See: kubeflow/kubeflow#5345

@ConverJens Thanks a lot! This solves for some of our use cases as well (i.e. deploying pipelines from shell)! Really appreciated.

@rafaelbarreto87 Happy to help!

Unfortunately, I think it would be great to have a more permanent and general solution for this (e.g. OAuth2) because some processes are not interactive and this method depend on the UI, which is something that could easily change. Either way, it's a workable solution for now, so thanks a lot again!

I agree. A more stable solution in necessary for the long term!

[yanp]

Here is a workaround to authenticate for kfp.Client() in Python:

import requests
import kfp

HOST = "http://localhost:8080/"
USERNAME = "[email protected]"
PASSWORD = "12341234"
NAMESPACE = "your-kf-ui-namespace"

session = requests.Session()
response = session.get(HOST)

headers = {
    "Content-Type": "application/x-www-form-urlencoded",
}

data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
session_cookie = session.cookies.get_dict()["authservice_session"]

client = kfp.Client(
    host=f"{HOST}/pipeline",
    cookies=f"authservice_session={session_cookie}",
    namespace=NAMESPACE,
)

print(client.list_pipelines())

[jlewi]

/cc @yanniszark @cventes

[tomalbrecht]

@yanp Thanks for this snippet. I've tested it, but I cannot get it to work yet. As I am using kfp from a remote client (mac -> ki-server3), I set kubectl edit svc authservice -n istio-system to NodePort on 30080. Thus I am able to reach the Pod, but no cookies are fetched (assuming NAMESPACE = 'kubeflow' for default installation).

Any ideas what I am doing wrong?

USERNAME = "[email protected]"
PASSWORD = "12341234"
#NAMESPACE = "your-kf-ui-namespace"
NAMESPACE = "kubeflow"   # "admin" does not work either
HOST = 'http://ki-server3:30080/'

session = requests.Session()
response = session.get(HOST)

headers = {
    "Content-Type": "application/x-www-form-urlencoded",
}

data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
print(session.cookies.get_dict())
session_cookie = session.cookies.get_dict()["authservice_session"]

client = kfp.Client(
    host=f"{HOST}/pipeline",
    cookies=f"authservice_session={session_cookie}",
    namespace=NAMESPACE,
)

print(client.list_pipelines())

Debugging Info:

# response
...
response.OK = True
response.reason = 'OK'
response.status_code = 200
...
# session
session.auth = None
session.cert = None
cookies = <RequestCookieJar[]>

In short RequestCookieJar[] is empty: {}

[yanp]

http://ki-server3:30080/

Hi @tomalbrecht, were you able to authenticate the Web UI on browser using url http://ki-server3:30080/ and USERNAME/PASSWORD above?

[tomalbrecht]

@yanp

Well, no. Not with USERNAME /PASSWORD. Port 30080 points to istio-system/authservice. When opening http://ki-server3:30080/ I get OKas an result. While opening https://ki-server3:31390 will point to dex and will ask for PASSWORD/USERNAME.

kubectl get svc -A | grep 8080
istio-system           authservice                                    NodePort       10.152.183.145   <none>        8080:30080/TCP                                                                                                                                                                                                                             6d12h

But this pointed me to the right direction. Thank you. I have to use the service istio-system/dex for authentication (https://ki-server3:31390/in my setup).

Now at least I get the cookie, but struggling with SSL certification in kfp.Client connecting from client -> server.

Traceback (most recent call last):
  File "/Users/tom/workspaces/ws-trevisto/stack/tests/wip_pipeline.py", line 144, in <module>
    print(client.list_pipelines())
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp/_client.py", line 458, in list_pipelines
    return self._pipelines_api.list_pipelines(page_token=page_token, page_size=page_size, sort_by=sort_by)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api/pipeline_service_api.py", line 1222, in list_pipelines
    return self.list_pipelines_with_http_info(**kwargs)  # noqa: E501
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api/pipeline_service_api.py", line 1327, in list_pipelines_with_http_info
    collection_formats=collection_formats)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 383, in call_api
    _preload_content, _request_timeout, _host)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 199, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 407, in request
    headers=headers)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/rest.py", line 248, in GET
    query_params=query_params)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/rest.py", line 226, in request
    headers=headers)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/request.py", line 76, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/poolmanager.py", line 330, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ki-server3', port=31390): Max retries exceeded with url: //pipeline/apis/v1beta1/pipelines?page_token=&page_size=10&sort_by= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Seems to be related to https://github.com/kubeflow/pipelines/issues/3095

[yanp]

@tomalbrecht I'm not sure about the ssl issue, but I had my port point to svc/istio-ingressgateway instead of dex, following the doc: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#accessing-kubeflow Hope it helps!

[tomalbrecht]

@yanp My fault. Port 31390 refers to svc/istio-ingressgateway and will forward to dex.

Did you change your gateway specs to use httpsRedirect: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#secure-with-https?

[yanp]

@tomalbrecht No I didn’t do any customization. I’d recommend asking for more help on the slack channel https://www.kubeflow.org/docs/about/community/#slack-community-and-channels

[yanp]

@tomalbrecht BTW namespace for client should be the one showing up on web UI, not necessarily kubeflow

[tomalbrecht]

For the records. Using http instead of ssl does work as a workaround on my microk8s manifest installation.

httpsRedirect has to be disabled:

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway
# set spec.hosts.tls.httpsRedirect: false

Then, the following code will work for me.

USERNAME = "[email protected]"
PASSWORD = "12341234"
NAMESPACE = "admin"
HOST = 'http://ki-server3:31380/'

session = requests.Session()
response = session.get(HOST)

headers = {
    "Content-Type": "application/x-www-form-urlencoded",
}

data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
print(session.cookies.get_dict())
session_cookie = session.cookies.get_dict()["authservice_session"]

print(session_cookie)

client = kfp.Client(
    host=f"{HOST}/pipeline",
    cookies=f"authservice_session={session_cookie}",
    namespace=NAMESPACE
)

print(client.list_pipelines())

Thanks for your help @yanp

[marcjimz]

+1 for all the helpful solutions to auth, thank you.

Any recommendations on how to do this in the reverse? Meaning once we have a cookie generated, is there anyway to validate that it is a valid cookie? For multi-user KfP I am able to verify this but only for a specific user or namespace, and am looking for general way to verify a cookie. Might be that this is flawed approach but wanted to put it in here.

[prashant2402]

I was able to authenticate following this flow. Not sure if it's the correct path but it worked for me using this installation https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/#add-static-users-for-basic-auth.

0) curl -v http://$SERVICE:$PORT
Response:
>> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=STATE_VALUE">Found</a>.

STATE=STATE_VALUE

1) curl -v "http://$SERVICE:$PORT/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&amp;state=$STATE_VALUE"
Response:
>> <a href="/dex/auth/local?req=REQ_VALUE">Found</a>

REQ=REQ_VALUE
2) curl -v 'http://$SERVICE:$PORT/dex/auth/local?req=REQ_VALUE' -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'

3) curl -v 'http://$SERVICE:$PORT/dex/approval?req=$REQ_VALUE'

Response:
>> <a href="/login/oidc?code=CODE_VALUE&amp;state=STATE_VALUE">See Other</a>.

CODE=CODE_VALUE

4) curl -v 'http://$SERVICE:$PORT/login/oidc?code=$CODE_VALUE&amp;state=$STATE_VALUE'

Response:
>> set cookie authservice_session=SESSION

curl -v 'http://$SERVICE:$PORT/pipeline/apis/v1beta1/pipelines' -H 'Cookie: authservice_session=SESSION'

Response:
>> 200 OK { ... }

Thanks @maurogonzalez for saving my day. this worked perfectly with my local kubeflow + dex in wsl2 converted into a nice little function to extract SESSION in my curl calls: If your endpoint is not localhost:8080 you can call the function with host and port args.

dex_session () {
        if [ $# -eq 0 ]; then HOST="localhost"; PORT="8080"; else HOST="$1"; PORT="$2";fi
        STATE=$(curl -s http://${HOST}:${PORT} | grep -oP '(?<=state=)[^ ]*"' | cut -d \" -f1)
        REQ=$(curl -s "http://${HOST}:${PORT}/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&amp;state=$STATE" | grep -oP '(?<=req=)\w+')
        curl -s "http://${HOST}:${PORT}/dex/auth/local?req=$REQ" -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'
        CODE=$(curl -s "http://${HOST}:${PORT}/dex/approval?req=$REQ" | grep -oP '(?<=code=)\w+')
        curl -s --cookie-jar - "http://${HOST}:${PORT}/login/oidc?code=$CODE&amp;state=$STATE" > .dex_session
        echo $(cat .dex_session | grep 'authservice_session' | awk '{print $NF}')
}

[benjamintanweihao]

Here's something that I came up with:

import logging
import os

import requests
from urllib.parse import parse_qs

host = 'https://your.inference.service'


def get_auth_session_token(host):
    response = requests.get(host, verify=False, allow_redirects=False)
    location = response.headers['location']

    state = parse_qs(location)['state'][0]

    location_prefix, _ = location.split('state=')
    location = location_prefix + f'state={state}'
    url = host + location

    response = requests.get(url, verify=False, allow_redirects=False)
    location = response.headers['location']
    req_value = location.split('=')[1]

    with requests.Session() as s:
        data = {
            'login': os.getenv('KUBEFLOW_USERNAME', 'EMAIL-GOES-HERE'),
            'password': os.getenv('KUBEFLOW_PASSWORD', 'PASSWORD-GOES-HERE'),
        }

        url = f'{host}/dex/auth/local?req={req_value}'
        logging.debug(url)

        response = requests.post(url, data=data, verify=False, allow_redirects=False)
        location = response.headers['location']
        req_value = location.split('=')[1]

        url = f'{host}/dex/approval?req={req_value}'
        logging.debug(url)
        response = requests.get(url, verify=False, allow_redirects=False)
        location = response.headers['location']

        url = host + location
        response = requests.get(url, verify=False, allow_redirects=False)
        logging.debug(url)

        return response.headers['set-cookie']

[LiuZhiYuan123456]

如何通过java进行dex认证，访问kubeflow集群呢？

[benjamintanweihao]

You can translate the Python code and do it in Java too. There's nothing specific to Python in the code. :)

On Thu, Nov 11, 2021 at 3:11 PM LiuZhiYuan123456 @.***> wrote:

如何通过java进行dex认证，访问kubeflow集群呢？

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubeflow/kfctl/issues/140#issuecomment-966046151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGSINAZC43RZKSPHV3GRELULNUAZANCNFSM4J3DGADQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.