Envoy Gateway AuthPolicy

[adam-cattermole]

Changes

• New controllers for envoy SecurityPolicies and ReferenceGrants
  • The SecurityPolicy enables external auth for envoy gateway
  • The ReferenceGrant permits access to SecurityPolicies that are not in the same namespace as the authorino authorization service (the kuadrant namespace)
• In this PR, we create a SecurityPolicy for each Gateway or Route targeted by an AuthPolicy, however the logic could be improved to create only for gateways that have no untargeted routes, and similarly, to not create one on a route, if all parent gateways have an attached AuthPolicy. However this current implementation is the simplest solution that does not rely on information outside of the remit of the controller based on the DAG.
• New controller for istio AuthorizationPolicy to allow the controller to be disabled when the CRDs are not present
• Added integration tests and modified workflows to run envoygateway integration tests

Verification

① Setup (Persona: Cluster admin)

make local-setup GATEWAYAPI_PROVIDER=envoygateway

Request an instance of Kuadrant in the kuadrant-system namespace:

kubectl -n kuadrant-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

② Deploy the Toy Store sample application (Persona: App developer)

kubectl apply -f examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: eg
    namespace: envoy-gateway-system
  hostnames:
  - api.toystore.com
  rules:
  - matches:
    - method: GET
      path:
        type: PathPrefix
        value: "/cars"
    - method: GET
      path:
        type: PathPrefix
        value: "/dolls"
    backendRefs:
    - name: toystore
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: "/admin"
    backendRefs:
    - name: toystore
      port: 80
EOF

Export the gateway hostname and port:

export INGRESS_HOST=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.status.addresses[0].value}')
export INGRESS_PORT=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.spec.listeners[?(@.name=="http")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Wait for the deployment:

kubectl wait --timeout=5m deployment/toystore --for=condition=Available

Send requests to the application unprotected:

curl -H 'Host: api.toystore.com' http://$GATEWAY_URL/cars -i
# HTTP/1.1 200 OK

curl -H 'Host: api.toystore.com' http://$GATEWAY_URL/dolls -i
# HTTP/1.1 200 OK

curl -H 'Host: api.toystore.com' http://$GATEWAY_URL/admin -i
# HTTP/1.1 200 OK

③ Protect the Toy Store application (Persona: App developer)

Create the AuthPolicy to enforce the following auth rules:

• Authentication:
  • All users must present a valid API key
• Authorization:
  • /admin* routes require user mapped to the admins group (kuadrant.io/groups=admins annotation added to the Kubernetes API key Secret)

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-authn":
        apiKey:
          selector: {}
        credentials:
          authorizationHeader:
            prefix: APIKEY
    authorization:
      "only-admins":
        opa:
          rego: |
            groups := split(object.get(input.auth.identity.metadata.annotations, "kuadrant.io/groups", ""), ",")
            allow { groups[_] == "admins" }
        routeSelectors:
        - matches:
          - path:
              type: PathPrefix
              value: "/admin"
EOF

Create the API keys:

kubectl apply -f -<<EOF
apiVersion: v1
kind: Secret
metadata:
  name: api-key-regular-user
  labels:
    authorino.kuadrant.io/managed-by: authorino
stringData:
  api_key: iamaregularuser
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: api-key-admin-user
  labels:
    authorino.kuadrant.io/managed-by: authorino
  annotations:
    kuadrant.io/groups: admins
stringData:
  api_key: iamanadmin
type: Opaque
EOF

Send requests to the application protected by Kuadrant:

curl -H 'Host: api.toystore.com' http://$GATEWAY_URL/cars -i
# HTTP/1.1 401 Unauthorized

curl -H 'Host: api.toystore.com' -H 'Authorization: APIKEY iamaregularuser' http://$GATEWAY_URL/cars -i
# HTTP/1.1 200 OK

curl -H 'Host: api.toystore.com' -H 'Authorization: APIKEY iamaregularuser' http://$GATEWAY_URL/admin -i
# HTTP/1.1 403 Forbidden

curl -H 'Host: api.toystore.com' -H 'Authorization: APIKEY iamanadmin' http://$GATEWAY_URL/admin -i
# HTTP/1.1 200 OK

④ Create a default "deny-all" policy at the level of the gateway (Persona: Platform engineer)

Create the policy:

kubectl -n envoy-gateway-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gw-auth
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
    authorization:
      deny-all:
        opa:
          rego: "allow = false"
    response:
      unauthorized:
        headers:
          "content-type":
            value: application/json
        body:
          value: |
            {
              "error": "Forbidden",
              "message": "Access denied by default by the gateway operator. If you are the administrator of the service, create a specific auth policy for the route."
            }
EOF

The policy won't be effective until there is at least one accepted route not yet protected by another more specific policy attached to it.

Create a route that will inherit the default policy attached to the gateway:

kubectl apply -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: other
spec:
  parentRefs:
  - name: eg
    namespace: envoy-gateway-system
  hostnames:
  - "*.other-apps.com"
EOF

Send requests to the route protected by the default policy set at the level of the gateway:

curl -H 'Host: foo.other-apps.com' http://$GATEWAY_URL/ -i
# HTTP/1.1 403 Forbidden

⑤ Check the new resources

Take a look at the SecurityPolicy and ReferenceGrant resources created for these auth policies:

kubectl get -n envoy-gateway-system securitypolicy/on-eg -o yaml | yq

kubectl get securitypolicy/on-toystore -o yaml | yq

kubectl get -n kuadrant-system referencegrant/kuadrant-authorization-rg -o yaml | yq

Cleanup

make local-cleanup

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 75.67568% with 144 lines in your changes missing coverage. Please review.

Please upload report for BASE (envoygateway@3dc9160). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
pkg/envoygateway/mutators.go	36.11%	14 Missing and 9 partials :warning:
...s/envoysecuritypolicy_referencegrant_controller.go	79.31%	11 Missing and 7 partials :warning:
...llers/authpolicy_envoysecuritypolicy_controller.go	88.37%	9 Missing and 6 partials :warning:
...authpolicy_istio_authorizationpolicy_controller.go	81.01%	10 Missing and 5 partials :warning:
...library/mappers/envoysecuritypolicy_to_kuadrant.go	37.50%	10 Missing and 5 partials :warning:
pkg/kuadranttools/topology_tools.go	77.77%	7 Missing and 7 partials :warning:
pkg/istio/mutators.go	45.83%	8 Missing and 5 partials :warning:
pkg/library/mappers/httproute_to_kuadrant.go	63.63%	8 Missing and 4 partials :warning:
pkg/library/mappers/gateway_to_kuadrant.go	62.50%	6 Missing and 3 partials :warning:
pkg/library/mappers/policy_to_kuadrant.go	57.14%	6 Missing and 3 partials :warning:
... and 1 more

Additional details and impacted files

@@               Coverage Diff               @@
##             envoygateway     #737   +/-   ##
===============================================
  Coverage                ?   82.02%           
===============================================
  Files                   ?       86           
  Lines                   ?     6616           
  Branches                ?        0           
===============================================
  Hits                    ?     5427           
  Misses                  ?      806           
  Partials                ?      383           

Flag	Coverage Δ
bare-k8s-integration	5.00% <8.44%> (?)
controllers-integration	73.84% <74.15%> (?)
envoygateway-integration	37.43% <55.91%> (?)
gatewayapi-integration	10.94% <8.44%> (?)
istio-integration	53.82% <27.87%> (?)
unit	29.56% <5.50%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	71.42% <0.00%> (?)
api/v1beta2 (u)	85.35% <0.00%> (?)
pkg/common (u)	88.13% <0.00%> (?)
pkg/istio (u)	71.11% <0.00%> (?)
pkg/log (u)	94.73% <0.00%> (?)
pkg/reconcilers (u)	∅ <0.00%> (?)
pkg/rlptools (u)	83.64% <0.00%> (?)
controllers (i)	83.37% <0.00%> (?)

Files with missing lines	Coverage Δ
controllers/authpolicy_controller.go	86.46% <ø> (ø)
controllers/test_common.go	100.00% <100.00%> (ø)
pkg/istio/mesh_config.go	67.51% <100.00%> (ø)
pkg/library/gatewayapi/topology.go	81.73% <100.00%> (ø)
pkg/library/gatewayapi/topology_indexes.go	93.06% <100.00%> (ø)
pkg/library/kuadrant/kuadrant.go	85.29% <ø> (ø)
pkg/rlptools/topology_index.go	74.35% <100.00%> (ø)
pkg/envoygateway/utils.go	88.88% <88.88%> (ø)
pkg/library/mappers/gateway_to_kuadrant.go	62.50% <62.50%> (ø)
pkg/library/mappers/policy_to_kuadrant.go	57.14% <57.14%> (ø)
... and 8 more

[adam-cattermole]

@eguzki and I discussed offline the idea of changing the controller to be written in terms of Kuadrants rather than for gateways, and instead iterate over the policies within the topology (with the new changes from main). We then can set the owner ref as the auth policy to handle deletion of the security policies. I've pushed a change for this in the latest commit (888f81d) but I still see two issues:

1. The topology is built in terms of the gateways and routes that are affected by which policies. This means however that if a SecurityPolicy exists for an AuthPolicy, for which the target (gateway/route) is deleted, the policy is filtered from the topology and so there is not a straightforward way to delete the SecurityPolicy (hence the failing tests)
2. The use of the owner ref as the AuthPolicy is assuming that there is one AuthPolicy per target. There is actually only one SecurityPolicy per target object which is why I set the owner like this before. If we have multiple AuthPolicies targeting the same object - which one should be the "owner"? and when the "owner" is deleted the SecurityPolicy will be removed even though there is still an AuthPolicy targeting the object.

In other words, I'm not sure the change in the latest commit (888f81d) actually solves any problems and is any more of an ideal solution than the previous version.

I'm curious if you have any thoughts/opinions on this @guicassolato?