Generalize Route CRD to be less Cloud Foundry-specific

[tcdowney]

tl;dr

Remove/rename some of the fields in the Route CRD to be less Cloud Foundry-specific.

Background

I want to use this Issue to share some thoughts a few of us have had about the Route CRD and to provide a place for discussion. I don't expect anything concrete to come out of this Issue specifically, but wanted to get some of these thoughts written down somewhere -- especially as y'all begin exploring alternative Ingress/mesh implementations which might require changes to the Route CRD.

The Route CRD (sample Route CR) has several fields that are specific to Cloud Foundry -- especially the app portion of a destination:

...

spec:
  destinations:
  - app:
      guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
      process:
        type: web

...

The spec.destinations[].selector field is what is ultimately used for configuring routing. The spec.destinations[].app field is used for providing a few helpful labels on the created Services and for setting headers that are used for app access logging.

This design has been useful since it closely mirrors the CF v3 Route API. Unfortunately, this design also tightly couples the Route with Cloud Foundry apps and makes it difficult to adapt to other (theoretical) types of routable workloads.

In addition to making it a more flexible interface, this has the added benefit of making the Route CR serve as an example for enhancements we might want to propose upstream.

For example, one day we may want to standardize on some of the next generation Kubernetes Service APIs -- perhaps even use the HTTPRoute/TCPRoute resources directly¹ -- and the CF Route CR could be something we use to drive out enhancements to that.

Or maybe this helps it becomes a more general ApplicationRoute that could be used in other PaaS-like contexts.

¹ - The HTTPRoute/TCPRoute may be too low-level and not something a component like the CF API would want to make directly. I'm mostly using these resources as a straw dog example for a standard we could work toward.

Proposal

App Guid / Process Type

Consider removing the required spec.destinations[].app attribute and replacing it with several optional attributes that more clearly describe the intent of the field.

• spec.destinations[].logging_id - Optional String identifier to use for aggregating logs associated with this route. In Cloud Foundry this would be used to tie access logs for a given route to the app they belong to. For Istio this would ultimately end up being a header that gets added to the request.
• spec.destinations[].headers - Optional map[string]string containing arbitrary headers to add/modify on the request.

Destination Guid

The spec.destinations[].guid attribute is a little strange. It is currently being relied on to give the child Services a unique name. A guid is suitable for this purpose, but it's a little strange that we are relying on the creator of the Route to know this and provide the guid. Perhaps we could rename this field to name (and in Cloud Foundry it's value could remain a guid) and the actual underlying child Services could be named <route-name>-<random-identifier>. Essentially the same as how a Deployment named cf-api-server would create a ReplicaSet named something like cf-api-server-c7d8b8f64.

• spec.destinations[].name - String containing the name of the destination

Domain

The spec.domain attribute is a little interesting. I'm not sure what the internal route UX will end up looking like, but this might be a field we could revisit as well. That said, it might be generic enough as is.

Metadata for Child Resources

The labels we were adding to the child Service created by each destination were more of a nice-to-have, but maybe there can be some form of template.metadata.labels/template.metadata.annotations that could propagate to child resources. I'm not going to include this on the proposed example since it's not strictly necessary, but still something to think about.

Example Route

With these changes a Route CR would look like:

apiVersion: networking.cloudfoundry.org/v1alpha1
kind: Route
metadata:
  name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
  namespace: cf-workloads
  labels:
    app.kubernetes.io/component: cf-networking
    app.kubernetes.io/managed-by: cloudfoundry
    app.kubernetes.io/name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
    app.kubernetes.io/part-of: cloudfoundry
    app.kubernetes.io/version: 0.0.0
    cloudfoundry.org/domain_guid: 23bb47a0-b042-4087-8e55-97ec4b69b43a
    cloudfoundry.org/org_guid: b7ab8526-b63b-4156-90b7-2cacfd686a8b
    cloudfoundry.org/route_guid: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
    cloudfoundry.org/space_guid: d4a93829-fed3-497a-bcba-00bb2d454681
spec:
  destinations:
    - name: 9363095c-6be5-4982-a7db-a493e74af2f4
      logging_id: be261513-3ccd-4000-b9d8-0023bbb08fbf
      headers:
        CF-App-Id: be261513-3ccd-4000-b9d8-0023bbb08fbf
        CF-App-Process-Type: web
        CF-Space-Id: d4a93829-fed3-497a-bcba-00bb2d454681
        CF-Organization-Id: b7ab8526-b63b-4156-90b7-2cacfd686a8b
      port: 8080
      selector:
        matchLabels:
          cloudfoundry.org/app_guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
          cloudfoundry.org/process_type: web
  domain:
    internal: false
    name: apps.example.com
  host: catnip
  path: ''
  url: catnip.apps.example.com

Just wanted to start an open discussion here to throw some of these ideas out there. If this is something y'all might actually do I'd be happy to collaborate on the design via Google Doc, PR, or other means.

cc @zrob who has also 🤔 about this and experimented with the Route CR