authzed
Document config format and template variables
As a workaround to comprehensive documentation,
the proxy rule source code defines the structure of the rules configuration (e.g. rules.yaml)
https://github.com/authzed/spicedb-kubeapi-proxy/blob/6c80fa7afb88dd8b2864234a2d76fd453f857ab6/pkg/config/proxyrule/rule.go#L22-L97
Must and MustNot are preconditions to be met before the transactions commits (in spice db and k8s), possibly to optimize or avoid inserting existing relationship, or ways to limit the scope of the rule.
the sample rules reproduced below provide examples of the rules format
https://github.com/authzed/spicedb-kubeapi-proxy/blob/6c80fa7afb88dd8b2864234a2d76fd453f857ab6/deploy/rules.yaml#L1-L88
The rules unit test source code details the templating syntax of the fields from the k8s resource
https://github.com/authzed/spicedb-kubeapi-proxy/blob/6c80fa7afb88dd8b2864234a2d76fd453f857ab6/pkg/config/proxyrule/rule_test.go#L102-L201
The templating jmespath expressions are also detailed in the pkg/rules/rules_test.go https://github.com/authzed/spicedb-kubeapi-proxy/blob/4745925492627fedeca235a041802f4a518797f8/pkg/rules/rules_test.go#L97-L134
The blog at https://authzed.com/blog/acl-filtering-kubernetes-apis-using-spicedb was a valuable complement to me
Getting started with the installation, by deploying the proxy from various methods
- Adjust sample deploy manifest available at https://github.com/authzed/spicedb-kubeapi-proxy/tree/main/deploy
- Kustomization bundles, see https://github.com/authzed/spicedb-kubeapi-proxy/releases/tag/v0.1.2
Include or modify this release in your own kustomize bundle
- Install with kustomizer https://github.com/authzed/spicedb-kubeapi-proxy/releases/tag/v0.1.2
Regarding usage of the proxy once deployed:
The E2E test shows the kubeconfig generation using X509 client certificates as user authN method:
https://github.com/authzed/spicedb-kubeapi-proxy/blob/6c80fa7afb88dd8b2864234a2d76fd453f857ab6/e2e/e2e_test.go#L296-L355
The client certs are already generated from the deployment at https://github.com/authzed/spicedb-kubeapi-proxy/blob/main/deploy/client-cert.yaml so the missing automation is the generation of a kubeconfig. Use of external secret operator templating would ease this kubeconfig generation.
Clients seem to be able to authenticate their users using multiple k8s standard user authentication methods, through usage of the same flags as the k8s api server flags documented into https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authentication-strategies :
- HTTP headers https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authenticating-proxy
The API server can be configured to identify users from request header values, such as X-Remote-User. It is designed for use in combination with an authenticating proxy, which sets the request header value.
https://github.com/authzed/spicedb-kubeapi-proxy/blob/188755c1fa7f139f8a66a736d80ce26e537c0916/pkg/proxy/authn.go#L25 https://github.com/authzed/spicedb-kubeapi-proxy/blob/0a7ead07d77467c977b191393da1fdf59ef07146/deploy/proxy.yaml#L41-L48 https://github.com/authzed/spicedb-kubeapi-proxy/blob/72a02a43ea2e9b69374785dd5bea11cdaf85e3ab/pkg/proxy/authn_test.go#L36-L47
- X509 client certificates https://kubernetes.io/docs/reference/access-authn-authz/authentication/#x509-client-certificates
If a client certificate is presented and verified, the common name of the subject is used as the user name for the request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships using the certificate's organization fields. To include multiple group memberships for a user, include multiple organization fields in the certificate.
https://github.com/authzed/spicedb-kubeapi-proxy/blob/2bb23917e1a712a01f4a950e39db15b42a60b301/e2e/e2e_test.go#L296-L298 https://github.com/authzed/spicedb-kubeapi-proxy/blob/188755c1fa7f139f8a66a736d80ce26e537c0916/pkg/proxy/authn.go#L20
- OpenID Connect Token https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
To identify the user, the authenticator uses the id_token (not the access_token) from the OAuth2 token response as a bearer token. https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-flags --oidc-username-claim JWT claim to use as the user name. By default sub [...] --oidc-groups-claim JWT claim to use as the user's group. [...]
https://github.com/authzed/spicedb-kubeapi-proxy/blob/188755c1fa7f139f8a66a736d80ce26e537c0916/pkg/proxy/authn.go#L21
- Static Token file https://kubernetes.io/docs/reference/access-authn-authz/authentication/#static-token-file
The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option on the command line. The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names. When using bearer token authentication from an http client, the API server expects an Authorization header with a value of Bearer
https://github.com/authzed/spicedb-kubeapi-proxy/blob/188755c1fa7f139f8a66a736d80ce26e537c0916/pkg/proxy/authn.go#L24
For convenience, here the usage output for version 0.1.2
docker run ghcr.io/authzed/spicedb-kubeapi-proxy:v0.1.2 -h
spicedb-kubeapi-proxy is a reverse proxy that authorizes Kube api
requests with SpiceDB and keeps relationship data up to date. The proxy handles
TLS termination and authentication with a backend kube apiserver.
Usage:
spicedb-kubeapi-proxy [flags]
Flags:
--api-audiences strings Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.
--backend-kubeconfig string The path to the kubeconfig to proxy connections to. It should authenticate the user with cluster-admin permission.
--bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces and IP address families will be used. (default 0.0.0.0)
--cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
--client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
--default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
-h, --help help for spicedb-kubeapi-proxy
--http2-max-streams-per-connection int The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log-json-info-buffer-size quantity [Alpha] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this.
--log-json-split-stream [Alpha] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this.
--logging-format string Sets the log format. Permitted formats: "json" (gated by LoggingBetaOptions), "text". (default "text")
--oidc-ca-file string If set, the OpenID server's certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host's root CA set will be used.
--oidc-client-id string The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.
--oidc-groups-claim string If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.
--oidc-groups-prefix string If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.
--oidc-issuer-url string The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).
--oidc-required-claim mapStringString A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims.
--oidc-signing-algs strings Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a supported 'alg' header values are: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, PS512. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1. (default [RS256])
--oidc-username-claim string The OpenID claim to use as the user name. Note that claims other than the default ('sub') is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details. (default "sub")
--oidc-username-prefix string If provided, all usernames will be prefixed with this value. If not provided, username claims other than 'email' are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value '-'.
--override-upstream if true, uses the environment to pick the upstream apiserver address instead of what is listed in --backend-kubeconfig. This simplifies kubeconfig management when running the proxy in the same cluster as the upstream. (default true)
--permit-address-sharing If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]
--permit-port-sharing If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]
--requestheader-allowed-names strings List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
--requestheader-extra-headers-prefix strings List of request header prefixes to inspect. X-Remote-Extra- is suggested.
--requestheader-group-headers strings List of request headers to inspect for groups. X-Remote-Group is suggested.
--requestheader-username-headers strings List of request headers to inspect for usernames. X-Remote-User is common.
--rule-config string The path to a file containing proxy rule configuration
--secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
--spicedb-ca-path string If set, looks in the given directory for CAs to trust when connecting to SpiceDB.
--spicedb-endpoint string Defines the endpoint endpoint to the SpiceDB authorizing proxy operations. if embedded:// is specified, an in memory ephemeral instance created. (default "localhost:50051")
--spicedb-insecure If set to true uses the insecure transport configuration for gRPC. Set to false by default.
--spicedb-skip-verify-ca If set to true backend certificate trust chain is not verified. Set to false by default.
--spicedb-token string Specifies the preshared key to use with the remote SpiceDB
--tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
--tls-cipher-suites strings Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_RC4_128_SHA.
--tls-min-version string Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
--tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
--token-auth-file string If set, the file that will be used to secure the secure port of the API server via token authentication.
--use-in-cluster-config if true, uses the local cluster as the upstream and gets the configuration from the environment.
-v, --v Level number for the log level verbosity (default 2)
--version version for spicedb-kubeapi-proxy
--vmodule pattern=N,... comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
--workflow-database-path string Path for the file representing the SQLite database used for the workflow engine. (default "/tmp/dtx.sqlite")
Care should be taken for boolean flags, see https://github.com/spf13/pflag?tab=readme-ov-file#command-line-flag-syntax
--flag // boolean flags, or flags with no option default values --flag x // only on flags without a default value --flag=x
So --override-upstream=false to turn off this boolean flag with a default true, (while --override-upstream false would turn it on)