Different solution for storing Terraform plans

[itamar-marom]

trafficstars

Storing the Terraform plan in a secret has two limitations:

• Large plans cannot be stored in secret due to the secret limit
• Storing those secrets (on a large scale) in the etcd can cause some performance issues

We should find a different store for that.

[chanwit]

Thank you @itamar-marom

We would use a similar configuration as backend for storing plans.

[tomhuang12]

We recently released a version that can configure a custom backend. Please let us know if this meets your needs.

[bivas]

@tomhuang12 this issue is for plan storage (binary and JSON format) not Terraform state (which covered by custom backend as you mentioned). Current plan storage is Kubernetes Secret and it's problematic with large plans (we have plans averaging at 6.5MB as JSON format) and a different storage is required (e.g. PVC)

[itamar-marom]

Hi @chanwit @tomhuang12

Our goal:

Be able to store Terraform plan files in persistent storage.

• Storage type will be agnostic
• Terraform Controller can be run locally

Our suggested solution:

Use Kubernetes’ PersistentVolume and PersistentVolumeClaim storage abstraction solutions. We want to release this solution in two steps.

For production, any supported StorageClass can be used.
A PVC with the EmptyDir option can be attached for a local run.

In each of those solutions, a PVC will be attached to the runner pod.

Version 1

Terraform will have a plan configuration field in its spec with a claimName field - a reference to a PVC to use. Terraform controller will then attach the PVC to the relevant runner pod.

Sample API:

# We create the PVC ourself
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: 
spec:
  planConfig: # Configuration related to Terraform plan
    storage:
      claimName: efs-claim # PVC name to use in runner
  sourceRef: # GitRepository as source
    kind: GitRepository
    name: helloworld
    namespace: flux-system

Version 2

Instead of user has to create the PVC on it's own, we would the Terraform controller to have a similar solution to Kubernetes StatefulSet with its volumeClaimTemplate spec. In this stage, the Terraform controller will create the PVC on its own for each runner.

Sample API: (We also talked about the possibility of making it more simple but less similar to StatefulSet's spec)

# StatefulSet style with volumeClaimTemplate
# https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#components
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: 
spec:
  planConfig: # Configuration related to Terraform plan
    volumeClaimTemplates:
      - metadata:
          name:
        spec:
          accessModes: [ "ReadWriteMany" ]
          storageClassName: efs-sc
          resources:
            requests:
              storage:
  sourceRef: # GitRepository as source
    kind: GitRepository
    name: helloworld
    namespace: flux-system

[lasomethingsomething]

Closing as a dupe of https://github.com/weaveworks/tf-controller/issues/536.