manifests
manifests copied to clipboard
kubeflow
How can I bypass Dex during inference?
In version 1.5 i was able to bypass Dex using extension provider. I added it to the istio configmap:
extensionProviders:
- name: "dex-auth-provider"
envoyExtAuthzHttp:
service: "authservice.istio-system.svc.cluster.local"
port: "8080"
includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
headersToUpstreamOnAllow: ["kubeflow-userid"]
Next I created two AuthorizationPolicy:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: dex-auth
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: CUSTOM
provider:
# The provider name must match the extension provider defined in the mesh config.
name: dex-auth-provider
rules:
# The rules specify when to trigger the external authorizer.
- to:
- operation:
notPaths: ["/v1*"]
and
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-inference-services
namespace: istio-system
spec:
selector:
matchLabels:
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
- to:
- operation:
methods: ["POST"]
paths: ["/v1*"]
Then I removed the filter and restarted istiod:
microk8s kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter
microk8s kubectl rollout restart deployment/istiod -n istio-system
After these manipulations, I could access Inference Services at the address http://host_ip:host_port/v1/models/model_name:predict without the need for authorization, since all requests from /v1* went bypassing Dex. Now I'm getting a 404 error.
It seems that requests are redirected to the central dashboard...
microk8s kubectl logs -n istio-system istio-ingressgateway-8f46b776-8sbt5
"POST /v1/models/my-model:predict HTTP/1.1" 404 - via_upstream - "-" 221 172 7 6 "192.168.21.45" "python-requests/2.31.0" "bf880dc0-918d-422b-a418-2d608de132bf" "my-model.my-namespace.example.com" "10.1.251.224:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.1.251.225:43688 10.1.251.225:8080 192.168.21.45:6542 - -
Request:
curl -v POST 'http://127.0.01:32333/v1/models/my-model:predict' -H 'Host:my-model.my-namespace.example.com' -d '{"instances": [{"data": {"req": ["My request"]}}]}'
How do I bypass Dex?
P.S. I can't get session.
This is what the log looks like in KubeFlow 1.5 with the Dex bypass described above:
[2023-11-22T06:00:46.148Z] "POST /v1/models/my-model:predict HTTP/1.1" 200 - via_upstream - "-" 221 777 8433 8433 "192.168.21.44" "python-requests/2.31.0" "835184ee-8830-92d5-a9ba-f8be36fe192a" "my-model-predictor-default.my-namespace.svc.cluster.local" "10.1.1.95:8081" outbound|80||knative-local-gateway.istio-system.svc.cluster.local 10.1.1.94:53100 10.1.1.94:8080 192.168.21.44:54926 - -
Requests go to Knative and not to Central Dashboard. How can I achieve the same behavior in version 1.8?
There is a difference in the output for the command kubectl get inferenceservices -n my-namespace
For version 1.5:
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
my-model http://my-model.my-namespace.example.com True 100 my-model -predictor-default-00001 3m56s
For version 1.8:
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
my-model http://my-model.my-namespace.svc.cluster.local True 100 my-model -predictor-00001 15s
In the first case, the service address ends with example.com, in the second with svc.cluster.local.
How can I perform inference for a deployed model?
There is a difference in the output for the command kubectl get virtualservices -n my-namespace
For version 1.5:
NAMESPACE NAME GATEWAYS HOSTS AGE
auth dex ["kubeflow/kubeflow-gateway"] ["*"] 23h
kubeflow centraldashboard ["kubeflow-gateway"] ["*"] 23h
kubeflow jupyter-web-app-jupyter-web-app ["kubeflow-gateway"] ["*"] 23h
kubeflow katib-ui ["kubeflow-gateway"] ["*"] 23h
kubeflow metadata-grpc ["kubeflow-gateway"] ["*"] 23h
kubeflow ml-pipeline-ui ["kubeflow-gateway"] ["*"] 23h
kubeflow profiles-kfam ["kubeflow-gateway"] ["*"] 23h
kubeflow tensorboards-web-app-tensorboards-web-app ["kubeflow-gateway"] ["*"] 23h
kubeflow volumes-web-app-volumes-web-app ["kubeflow-gateway"] ["*"] 23h
kubeflow kserve-models-web-app ["kubeflow/kubeflow-gateway"] ["*"] 23h
my-namespace notebook-my-namespace-megaputer-notebook ["kubeflow/kubeflow-gateway"] ["*"] 23h
my-namespace my-model-predictor-default-mesh ["mesh"] ["my-model-predictor-default.my-namespace","my-model-predictor-default.my-namespace.svc","my-model-predictor-default.my-namespace.svc.cluster.local"] 22h
my-namespace my-model-predictor-default-ingress ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"] ["my-model-predictor-default.my-namespace","my-model-predictor-default.my-namespace.example.com","my-model-predictor-default.my-namespace.svc","my-model-predictor-default.my-namespace.svc.cluster.local"] 22h
my-namespace my-model ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"] ["my-model.my-namespace.svc.cluster.local","my-model.my-namespace.example.com"] 22h
For version 1.8:
NAMESPACE NAME GATEWAYS HOSTS AGE
auth dex ["kubeflow/kubeflow-gateway"] ["*"] 22h
kubeflow centraldashboard ["kubeflow-gateway"] ["*"] 22h
kubeflow jupyter-web-app-jupyter-web-app ["kubeflow-gateway"] ["*"] 22h
kubeflow katib-ui ["kubeflow-gateway"] ["*"] 22h
kubeflow metadata-grpc ["kubeflow-gateway"] ["*"] 22h
kubeflow ml-pipeline-ui ["kubeflow-gateway"] ["*"] 22h
kubeflow profiles-kfam ["kubeflow-gateway"] ["*"] 22h
kubeflow tensorboards-web-app-tensorboards-web-app ["kubeflow-gateway"] ["*"] 22h
kubeflow volumes-web-app-volumes-web-app ["kubeflow-gateway"] ["*"] 22h
kubeflow kserve-models-web-app ["kubeflow/kubeflow-gateway"] ["*"] 22h
my-namespace notebook-my-namespace-megaputer-notebook ["kubeflow/kubeflow-gateway"] ["*"] 21h
my-namespace my-model-predictor-mesh ["mesh"] ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"] 19h
my-namespace my-model-predictor-ingress ["knative-serving/knative-local-gateway"] ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"] 19h
my-namespace my-model ["knative-serving/knative-local-gateway"] ["my-model.my-namespace.svc.cluster.local"] 19h
No kubeflow/kubeflow-gateway specified for version 1.8. How can this be fixed?
@juliusvonkohout, it seems impossible to do inference from the outside. Virtual Services of Inference Services do not contain a gateway looking out (kubeflow-gateway).
I'm not sure, but maybe it's a missing line local-gateway.mesh: "mesh" in the file net-istio.yaml.
Can you join the next manifest WG meeting for discussion? It has been possible before. Can you use path based routing? Why do you not want the oidc-authservice token based authorization?
Yes, I can join the discussion. Previously (in KubeFlow 1.5) this was possible and I used it. The fact is that a large number of requests end up in the Dex queue, which ultimately breaks it and the cluster. Most likely the problem is not with Dex bypass, but with the inability to perform Inference from outside the cluster. I think this issue is related to this.
Any conclusion to this issue? Looking for the exact same solution where I want to bypass the oidc authuservice when exposing the Inference service to outside the cluster while making use of istio. Facing the same issue where the requests end up going to central dashboard.
Patching the knative configmap config-domain with a custom domain atleast updates the virtual service so that it makes use of the kubeflow-gateway for external traffic.
Got it to work with the authentication where I could pass the Cookie authservice_session. But would be better if we could have an option to bypass this.
Unfortunately, I was never able to beat Dex in version 1.8. In addition, I was never able to access the InferenceService from the outside.
@ksgnextuple, are you able to at least access InferenceService with a session?
@ksgnextuple, сan you share the output of the following command?
kubectl get virtualservices -n my-namespace
@yurkoff-mv I am not using dex, but the oidc-uservice.
Here is the output:
sklearn-v2-iris-predictor-ingress ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"] ["sklearn-v2-iris-predictor-dev.example.com","sklearn-v2-iris-predictor.dev","sklearn-v2-iris-predictor.dev.svc","sklearn-v2-iris-predictor.dev.svc.cluster.local"] 5d20h
sklearn-v2-iris-predictor-mesh ["mesh"] ["sklearn-v2-iris-predictor.dev","sklearn-v2-iris-predictor.dev.svc","sklearn-v2-iris-predictor.dev.svc.cluster.local"] 5d20h
I also had to update knative config domain, that was the key
kind: ConfigMap
metadata:
name: config-domain
namespace: knative-serving
labels:
app.kubernetes.io/name: knative-serving
app.kubernetes.io/component: controller
app.kubernetes.io/version: "1.10.2"
annotations:
knative.dev/example-checksum: "26c09de5"
data:
example.com: ""
_example: | ```
@yurkoff-mv Which ConfigMap are you adding the extension provider? I haven't tried the steps you had mentioned for bypassing auth on my setup yet
@yurkoff-mv Was able to achieve it. Disabled session token as well for Infer endpoints. Though not recommended for a production setup. We wanted it for our nonprod env.
Thank you! It's interesting that kubeflow/kubeflow-gateway doesn't appear in my VirtualService.
@yurkoff-mv Which ConfigMap are you adding the extension provider? I haven't tried the steps you had mentioned for bypassing auth on my setup yet
I didn't change the ConfigMap
@yurkoff-mv Was able to achieve it. Disabled session token as well for Infer endpoints. Though not recommended for a production setup. We wanted it for our nonprod env.
How did you disable the session token?
@yurkoff-mv The same steps you had mentioned above for disabling the session token. For the kubeflow-gateway to come up the knative configmap needs to be updated in the way I had mentioned. Reference on Knative website -> https://knative.dev/docs/serving/using-a-custom-domain/
https://kserve.github.io/website/0.10/admin/serverless/serverless/#1-install-knative-serving - Look at the warning here too. The changes was mostly from knative side
@ksgnextuple, thank you very much! It worked for me!
It's strange that this behavior is not enabled by default.
For future readers, I will write that in the source code of the inference it is worth replacing authservice_session with oauth2_proxy_kubeflow.
It's strange that these steps are not described in this repository. My problem has been hanging around for several months now and only you could help me. Thank you again!
Now I will try to bypass authorization for the inference service.
@ksgnextuple, Could you please provide your ConfigMap for clarity?
kubectl edit configmap istio -n istio-system
data:
mesh: |-
extensionProviders:
- name: "oidc-auth-provider"
envoyExtAuthzHttp:
service: "authservice.istio-system.svc.cluster.local"
port: "8080"
includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
headersToUpstreamOnAllow: ["kubeflow-userid"]
accessLogFile: /dev/stdout
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing: {}
enablePrometheusMerge: false
rootNamespace: istio-system
tcpKeepalive:
interval: 5s
probes: 3
time: 10s
trustDomain: cluster.local
meshNetworks: 'networks: {}'
kind: ConfigMap
It’s similar for me. Only when I try to perform inference I get RBAC: access denied.
curl -v -H "Host: sklearn-iris.kubeflow-user-example-com.example.com" -H "Content-Type: application/json"
"http://localhost:32333/v1/models/sklearn-iris:predict" -d @./iris-input.json
* Trying 127.0.0.1:32333...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 32333 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
>
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 19
< content-type: text/plain
< date: Thu, 04 Apr 2024 06:33:56 GMT
< server: istio-envoy
< connection: close
<
* Closing connection 0
RBAC: access denied
It’s also strange that there was no EnvoyFilter authn-filter by default.
I had to remove the authn-filter Envoy filter. This RBAC access denied i think seems to be triggered by the global deny auth policy. Not too sure though.
@ksgnextuple, thank you for the help.
It didn’t work for me because by default OAuth2-proxy is used as authorization instead of OIDC AuthService. The instructions given at the beginning of the topic work for OIDC AuthService. Therefore, you need to either immediately deploy KubeFlow with OIDC AuthService, or think about how to bypass OAuth2-proxy. There is no EnvoyFilter authn-filter in OAuth2-proxy. In theory, it would be enough to deploy the two above AuthorizationPolicies. Only in the first one replace dex-auth-provider with oauth2-proxy. But for some reason this doesn't work.
This is what the Extension Provider looks like for OAuth2-proxy:
extensionProviders:
- envoyExtAuthzHttp:
headersToDownstreamOnDeny:
- content-type
- set-cookie
headersToUpstreamOnAllow:
- authorization
- path
- x-auth-request-email
- x-auth-request-groups
- x-auth-request-user
includeRequestHeadersInCheck:
- authorization
- cookie
service: oauth2-proxy.oauth2-proxy.svc.cluster.local
port: 80
name: oauth2-proxy
Its definition can be found in the following yaml file.
Yes, I did it!
As a result, with OAuth2-proxy it was even easier to bypass Dex...
You just need to edit AuthorizationPolicy istio-ingressgateway-oauth2-proxy, adding rules to it:
kubectl edit AuthorizationPolicy -n istio-system istio-ingressgateway-oauth2-proxy
It should look like this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
- to:
- operation:
notPaths: ["/v1*"]
For testing in Python:
KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333" # Cluster IP and port
KUBEFLOW_USERNAME = "[email protected]"
KUBEFLOW_PASSWORD = "12341234"
MODEL_NAME = "sklearn-iris"
SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com"
PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict"
iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
res = requests.post(
url=PREDICT_ENDPOINT,
headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json"},
# cookies=jar,
json=iris_input,
timeout=200,
)
print("Status Code: ", res.status_code)
print("Response: ", res.json())
Output:
Status Code: 200
Response: {'predictions': [1, 1]}
Alternative for curl in console...
Create file iris-input.json:
cat <<EOF > "./iris-input.json"
{
"instances": [
[6.8, 2.8, 4.8, 1.4],
[6.0, 3.4, 4.5, 1.6]
]
}
EOF
Request:
curl -v -H "Host: sklearn-iris.kubeflow-user-example-com.example.com" -H "Content-Type: application/json"
Response:
"http://127.0.0.1:32333/v1/models/sklearn-iris:predict" -d @./iris-input.json
* Trying 127.0.0.1:32333...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 32333 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
>
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 21
< content-type: application/json
< date: Fri, 05 Apr 2024 05:22:58 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 4
<
* Connection #0 to host 127.0.0.1 left intact
{"predictions":[1,1]}
@yurkoff-mv @ksgnextuple this looks more like introducing a security flaw than using proper tokens from serviceaccounts with oauth2-proxy. I am very open to merge something with proper authentication.
CC @kromanow94
@juliusvonkohout It was actually just more of a 'is it possible scenario' to bypass auth, ideally not recommended off course. And the changes we had to do were at auth, Isio and Knative level.
I agree with Julius that this is making the setup less secure but if you really want to do this, the AuthorizationPolicy provided in this comment https://github.com/kubeflow/manifests/issues/2575#issuecomment-2038972474 would be the way to go. In general, using AuthorizationPolicies is the preferred way of configuring authorization with the recent changes for Authz introduced in https://github.com/kubeflow/manifests/pull/2544.
But, please consider using M2M tokens, functionality natively integrated with oauth2-proxy/istio, also configured as part of latest changes around Authz.
Considering the above example in python, it should be enough to do:
export TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
export AUTHORIZATION = "Bearer: ${TOKEN}"
AUTHORIZATION = os.getenv("AUTHORIZATION")
KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333" # Cluster IP and port
MODEL_NAME = "sklearn-iris"
SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com"
PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict"
iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
res = requests.post(
url=PREDICT_ENDPOINT,
headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json", "Authorization": AUTHORIZATION},
json=iris_input,
timeout=200,
)
print("Status Code: ", res.status_code)
print("Response: ", res.json())
With the above, you should be able to just use the latest master version of kubeflow/manifests, most probably without additional changes around istio and knative. I welcome you to try and provide feedback in this setup.
I agree with Julius that this is making the setup less secure but if you really want to do this, the AuthorizationPolicy provided in this comment #2575 (comment) would be the way to go. In general, using AuthorizationPolicies is the preferred way of configuring authorization with the recent changes for Authz introduced in #2544.
But, please consider using M2M tokens, functionality natively integrated with oauth2-proxy/istio, also configured as part of latest changes around Authz.
Considering the above example in python, it should be enough to do:
export TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)" export AUTHORIZATION = "Bearer: ${TOKEN}"AUTHORIZATION = os.getenv("AUTHORIZATION") KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333" # Cluster IP and port MODEL_NAME = "sklearn-iris" SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com" PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict" iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]} res = requests.post( url=PREDICT_ENDPOINT, headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json", "Authorization": AUTHORIZATION}, json=iris_input, timeout=200, ) print("Status Code: ", res.status_code) print("Response: ", res.json())With the above, you should be able to just use the latest
masterversion ofkubeflow/manifests, most probably without additional changes around istio and knative. I welcome you to try and provide feedback in this setup.
@diegolovison is this something for the /contrib/kserve documentation?