cf-k8s-networking
cf-k8s-networking copied to clipboard
Generalize Route CRD to be less Cloud Foundry-specific
tl;dr
Remove/rename some of the fields in the Route CRD to be less Cloud Foundry-specific.
Background
I want to use this Issue to share some thoughts a few of us have had about the Route CRD and to provide a place for discussion. I don't expect anything concrete to come out of this Issue specifically, but wanted to get some of these thoughts written down somewhere -- especially as y'all begin exploring alternative Ingress/mesh implementations which might require changes to the Route CRD.
The Route CRD (sample Route
CR) has several fields that are specific to Cloud Foundry -- especially the app
portion of a destination
:
...
spec:
destinations:
- app:
guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
process:
type: web
...
The spec.destinations[].selector
field is what is ultimately used for configuring routing. The spec.destinations[].app
field is used for providing a few helpful labels on the created Services
and for setting headers that are used for app access logging.
This design has been useful since it closely mirrors the CF v3 Route API. Unfortunately, this design also tightly couples the Route
with Cloud Foundry apps and makes it difficult to adapt to other (theoretical) types of routable workloads.
In addition to making it a more flexible interface, this has the added benefit of making the Route
CR serve as an example for enhancements we might want to propose upstream.
For example, one day we may want to standardize on some of the next generation Kubernetes Service APIs -- perhaps even use the HTTPRoute
/TCPRoute
resources directly1 -- and the CF Route
CR could be something we use to drive out enhancements to that.
Or maybe this helps it becomes a more general ApplicationRoute
that could be used in other PaaS-like contexts.
1 - The HTTPRoute
/TCPRoute
may be too low-level and not something a component like the CF API would want to make directly. I'm mostly using these resources as a straw dog example for a standard we could work toward.
Proposal
App Guid / Process Type
Consider removing the required spec.destinations[].app
attribute and replacing it with several optional attributes that more clearly describe the intent of the field.
-
spec.destinations[].logging_id
- OptionalString
identifier to use for aggregating logs associated with this route. In Cloud Foundry this would be used to tie access logs for a given route to the app they belong to. For Istio this would ultimately end up being a header that gets added to the request. -
spec.destinations[].headers
- Optionalmap[string]string
containing arbitrary headers to add/modify on the request.
Destination Guid
The spec.destinations[].guid
attribute is a little strange. It is currently being relied on to give the child Services
a unique name. A guid is suitable for this purpose, but it's a little strange that we are relying on the creator of the Route
to know this and provide the guid. Perhaps we could rename this field to name
(and in Cloud Foundry it's value could remain a guid) and the actual underlying child Services
could be named <route-name>-<random-identifier>
. Essentially the same as how a Deployment
named cf-api-server
would create a ReplicaSet
named something like cf-api-server-c7d8b8f64
.
-
spec.destinations[].name
-String
containing the name of the destination
Domain
The spec.domain
attribute is a little interesting. I'm not sure what the internal
route UX will end up looking like, but this might be a field we could revisit as well. That said, it might be generic enough as is.
Metadata for Child Resources
The labels we were adding to the child Service
created by each destination were more of a nice-to-have, but maybe there can be some form of template.metadata.labels
/template.metadata.annotations
that could propagate to child resources. I'm not going to include this on the proposed example since it's not strictly necessary, but still something to think about.
Example Route
With these changes a Route CR would look like:
apiVersion: networking.cloudfoundry.org/v1alpha1
kind: Route
metadata:
name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
namespace: cf-workloads
labels:
app.kubernetes.io/component: cf-networking
app.kubernetes.io/managed-by: cloudfoundry
app.kubernetes.io/name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
app.kubernetes.io/part-of: cloudfoundry
app.kubernetes.io/version: 0.0.0
cloudfoundry.org/domain_guid: 23bb47a0-b042-4087-8e55-97ec4b69b43a
cloudfoundry.org/org_guid: b7ab8526-b63b-4156-90b7-2cacfd686a8b
cloudfoundry.org/route_guid: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
cloudfoundry.org/space_guid: d4a93829-fed3-497a-bcba-00bb2d454681
spec:
destinations:
- name: 9363095c-6be5-4982-a7db-a493e74af2f4
logging_id: be261513-3ccd-4000-b9d8-0023bbb08fbf
headers:
CF-App-Id: be261513-3ccd-4000-b9d8-0023bbb08fbf
CF-App-Process-Type: web
CF-Space-Id: d4a93829-fed3-497a-bcba-00bb2d454681
CF-Organization-Id: b7ab8526-b63b-4156-90b7-2cacfd686a8b
port: 8080
selector:
matchLabels:
cloudfoundry.org/app_guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
cloudfoundry.org/process_type: web
domain:
internal: false
name: apps.example.com
host: catnip
path: ''
url: catnip.apps.example.com
Just wanted to start an open discussion here to throw some of these ideas out there. If this is something y'all might actually do I'd be happy to collaborate on the design via Google Doc, PR, or other means.
cc @zrob who has also 🤔 about this and experimented with the Route
CR