Expose bundle outputs inside a cluster

[carolynvs]

trafficstars

The outputs generated from running an installation should be natively accessible from Kubernetes. So after the installation completes the output values should be copied into a secret.

We have a couple options for output syncing:

1. Create a new secret for the outputs after each run. This more faithfully represents how porter works (outputs are immutable and tied to a porter run), but puts more decisions on the consumer. Such as do they want the most recent value of an output, or do they want the output generated by a specific generation of the installation.
2. Keep a single secret per installation and update it with the outputs after each run. This would be easier for people to consume but perhaps oversimplifies things.

I have added https://github.com/getporter/porter/issues/1877 to porter so that we can correlate an installation spec generation with a particular run (and its outputs).

[bdegeeter]

Based on our last dev meeting here's some high level ideas regarding outputs. Just wanted to get enough ideas out to go a bit deeper into the design.

• Annotation to enable

  • porter.sh/enable-outputs: "true"

  apiVersion: porter.sh/v1
  kind: Installation
  metadata:
    name: test-me
    namespace: quickstart
    annotations:
      # enable creation of InstallationOutput resource
      porter.sh/enable-outputs: "true"
  spec:
    schemaVersion: 1.0.0
    namespace: quickstart
    name: test-me
    bundle:
      repository: ghcr.io/bdegeeter/porter-test-me
      version: 0.3.0
    parameters:
      name: quickstart
  

• InstallationOutput CRD

  apiVersion: porter.sh/v1
  kind: InstallationOutput
  metadata:
    name: test-me # use the same name as the installation resource
    namespace: quickstart
    generation: 1
    labels:
      porter.sh/managed: "true"
      porter.sh/resourceGeneration: "1"
      porter.sh/resourceKind: Installation
      porter.sh/resourceName: test-me
    ownerReferences:
    - apiVersion: porter.sh/v1
      blockOwnerDeletion: true
      controller: true
      kind: Installation
      name: test-me
  spec:
    namespace: quickstart
    name: test-me
  status:
    # Should we use phase to show output is 
    # associated with an Installation in transition?
    phase: (Scheduled|Started|Completed|Succeeded)
    # all outputs from bundle
    outputs:
    - name: outAction
      type: string
      sensitive: false
      value: install
    - name: outDelay
      type: integer
      sensitive: false
      value: 1
    - name: outExitStatus
      type: integer
      sensitive: false
      value: install
    - name: outInsecureValue
      type: string
      sensitive: true
      key: plugin-secret-key-name
  

  • Notes

    • Outputs as status (taken directly from porter installation outputs show schema)
    • InstallationOutput as new resource for isolated RBAC
    • carry forward labels where appropriate
    • what are the limits of storing values in status?

  • Behaviors

    • Uses porter v1.0.0-alpha.20 behavior for outputs
      • If bundle does not mark output as sensitive (default) provide the value
      • If bundle marks output as sensitive provide the secret key
    • The InstallationOutput resource is created when Installation state Completed is discovered but before it is set.
      • goal is to guarantee output resource available when Installation complete
    • Should controller remove (or invalidate) existing InstallationOutputs? when?
      • Installation uninstall via resource attribute
        • remove or invalidate at the start of uninstall (with delete of InstallationOutput at end?)
      • Installation resource delete
        • remove or invalidate at the start of uninstall (with delete of InstallationOutput at end?)
      • Installation resource update for upgrade
        • remove or invalidate at the start of update (with update of resource at the end?)
      • Can a spec field or status be used to show an InstallationOutput is in transition because of an upgrade or uninstall?

[carolynvs]

This is great! Here are a few of my notes from today's community meeting where we discussed this:

• Let's associate the output CRDs to the AgentAction (i.e. the run that generated them) instead of the installation
• For now we should probably focus on "here are the outputs from a specific run" vs. "here is what we think is the latest output" due to risks and odd edge cases around the definition of "latest". Long term the way I recommend getting the "latest" output value is through querying porter directly.
• Move the collect outputs setting from an annotation to the AgentConfig

[bdegeeter]

We'll also be looking for a solution collect the outputs without extra costs of additional pod execution

[bdegeeter]

@carolynvs, what do you think about adding a feature to porter that writes outputs to a file as part of an action?

porter install porter-hello --reference ghcr.io/getporter/examples/porter-hello:v0.2.0 --outputs-file=/porter-outputs/outputs.json -o json

Then we can get the outputs in a volume the AgentAction controller manages with a single porter execution.

[carolynvs]

I don't think that is necessary. Right now with porter installation outputs list INSTALLATION -o yaml > /some/file we can can get all the outputs from an installation.

Let's look instead at tweaking the agent so that it can accept a list of commands to run. Then we can have the agent run install, then a second command to collect output. That will be more extensible and not muck with the command design that we have just to help us shoehorn this in. 😀

[carolynvs]

After talking with Steven at the porter community meeting today, we've realized that all the techniques for extracting the output value from a porter run are poor solutions that we don't want to implement.

There are problems with the timing of when pods are available, accessing a volume requires a pod (so we either have to spin one up, or use side-cars, etc), or use tricks to keep pods around longer than they normally persist, etc.

What we really need is the grpc service, so that the operator itself can query for output values and put them into CRDs.

[sgettys]

Another option I forgot to bring up is setting up a porter client in the controller and potentially calling the outputs directly from there although that also feels like a hacky workaround that would go away with a grpc service

[carolynvs]

oof, yeah I really don't want to put porter in the operator. Still voting for moving immediately to a grpc service! 😀

[carolynvs]

For operator v1, let's do a tiny grpc service that just exposes an endpoint for retriving outputs, and we can ship that with the operator. But the grpc service will continue to grow and expose more endpoints over time.

[bdegeeter]

Regarding gRPC service, what if a gRPC server mode was an option for the porter cli? We could then run it as an additional service along side the operator-manager. This cli mode could then start to expose commands starting with installation outputs. This could then be decoupled from an external facing API which would allow for graphQL or REST as needed to reduce tight api version coupling.

[carolynvs]

If we do that, let's make that command top secret so that we have some time to tinker without committing to exposing the command directly on the porter CLI using whatever protocol we try first. Possibly using conditional compilation so that only the porter agent has the extra command, but not the binary we distribute as the client.