cf-ops-automation icon indicating copy to clipboard operation
cf-ops-automation copied to clipboard

add k8s support

Open poblin-orange opened this issue 7 years ago • 7 comments

As cloudfoundry now supports Kubernetes with CloudFoundry Container Runtime, COA should offer first class support

  • [x] add k8s terraform plugin, so we can apply k8s config from terraform
  • [x] add helm chart support via terraform plugin
  • [ ] enable k8s containers provisionning (eg k8s-apps-depls naming convention, use a declarative format like helm, add kubctl to the pre / post scripts hosting containers)

cc @obeyler

poblin-orange avatar Feb 12 '18 09:02 poblin-orange

Including helm chart requires investigations, 2 providers exist:

  • https://github.com/mcuadros/terraform-provider-helm with some drawbacks preventing it to be an official one: https://github.com/hashicorp/terraform/issues/14126
  • https://github.com/djhaskin987/terraform-provider-helmcmd: addtionnal requirements (kubectl and helm binaries) and not yet used by the community

o-orand avatar Apr 17 '18 13:04 o-orand

mcuadros provider seem to have the biggest user community support. Seems the way to go short term

poblin-orange avatar Apr 17 '18 16:04 poblin-orange

Helm is now part of the base image: https://github.com/orange-cloudfoundry/paas-docker-cloudfoundry-tools/commit/ad445d6b34dffeadb3c2b26a40dd71de73ec0686. But not yet used by COA.

o-orand avatar Apr 23 '18 12:04 o-orand

Base image is now used by COA (from https://github.com/orange-cloudfoundry/cf-ops-automation/commit/27e680550bae917c2e3d8e613b3033a39ce48162) and will be part of next release (2.0.0)

o-orand avatar Apr 24 '18 07:04 o-orand

Merging #180 into this issue

gberche-orange avatar Aug 01 '18 12:08 gberche-orange

Requirements:

  • install helm chart
  • validate/test the chart by running associated tests if any, see chart_tests.md for background

The considered Tf provider is https://github.com/mcuadros/terraform-provider-helm

Steps:

  • install the provider
  • set up the provider prereq in the docker image: kubectl + helm init (as a workaround for https://github.com/mcuadros/terraform-provider-helm/issues/23)

Remaining issues:

  • the chart repository needs to be versioned and fresh. The concourse workers are stateless. When using the helm_repository resource, the set up repository is ephemeral. The format of the chart repository and its content is documented into https://github.com/helm/helm/blob/master/docs/chart_repository.md which complements charts detailed structure at https://github.com/helm/helm/blob/master/docs/charts.md

Possible workarounds:

  • have the provider support a new chart_repository data source, so that the chart repository is donwloaded systematically
  • have COA persist the chart repository directory
    • into the paas-secret git repo
    • by syncing the container with an external store prior and after the TF plan/apply command
  • don't use the helm_repository resource and rather execute helm commands prior to the TF execution (that will only apply the "helm_release" resource). TBD how

/CC @poblin-orange @obeyler

gberche-orange avatar Aug 01 '18 12:08 gberche-orange

Draft issue on https://github.com/mcuadros/terraform-provider-helm/ to suggest datasource.

@obeyler we need to record traces and behavior of the provider terraform-provider-helm when the helm_repository files are reported as missing

Issue title: Add support for a helm_repository datasource

### Actual Behavior

Currently the [helm_repository](https://github.com/mcuadros/terraform-provider-helm/blob/master/docs/repository.md) resources downloads the repository on local disk in `$HELM_HOME` and expects it to be persistent across TF executions

When this provider is used within stateless continuous integration systems, the local disk is ephemeral (along with any downloaded repository), so the files associated to the `helm_repository` are missing at each Tf plan/apply execution, and gets recreated

In addition, plain CI systems need the local environment setup as described in #23 



### Expected Behavior

Similarly to the [aws_s3_bucket_object](https://www.terraform.io/docs/providers/aws/d/s3_bucket_object.html) which provides a way to download a S3 bucket content in memory and use it from other resource, a `helm_repository` data source would systematically download the repository locally (in memory or local ephemeral disk).

This way stateless CI/CD systems can merely add static helm configuration, and don't need persist changes make to $HELM_HOME by the `terraform-provider-helm`

gberche-orange avatar Aug 08 '18 07:08 gberche-orange