rfc(template): remove custom Jinja templating

[shikanime]

As Reana uses Helm to install Traefik, I was thinking why not use Helm for back-end kubernetes instead of using custom templates with jinja. Since Helm is an industry standard for reusable Kubernetes packages, it can open up opportunities for independent deployment without a reana-cluster and allow the use of packages such as stable/postgresql.

I look forward to contributing more 🥰

[tiborsimko]

Hi @Shikanime, thanks for willingness to contribute!

We created reana-cluster in the old days so that we could deploy REANA in a plethora of ways, using Magnum/Mesos/whatnot. But since those days we have been focusing essentially on Kubernetes, so going Helm makes perfect sense.

We have also been looking at Kustomize which comes for free with kubectl and which would also avoid some Tiller related concerns... However I agree Helm is pretty much the standard these days and Kustomize dev/qa/prod related handling could be separate from Helm.

Note that @BenGalewsky already started to create REANA charts some moths ago:

• https://github.com/BenGalewsky/reana-cluster/tree/helm_chart/reana

Perhaps we could join the efforts and make the change happen?

[shikanime]

Hi @tiborsimko!

Speaking of Helm's Tiller, there have been several attempts in the community like the Operator Framework Helm Operator or bitnamy's Helm CRD. Well, fortunately the next major version 3 of Helm has definitely got rid of it. With that the security model for Helm is radically simplified.

Of course I'm up for it.

[pcasteran]

Indeed Helm's current model require special care when used on a Kubernetes cluster with Role-based access control (RBAC) enabled, which is the case for example on Google's managed clusters (GKE). Following Helm's official documentation on RBAC, some Kubernetes resources (service account, cluster role and binding, ...) are required and must be created before deploying Tiller.

If you are interested, I can submit the configuration required for Helm to deploy Reana in a RBAC-enabled cluster.

[shikanime]

I had a lot to do since then, I'm working on it right now 😇

[shikanime]

After some experimentation, Helm turns out not to be a suitable tool. It makes the YAML less readable, less easy to maintain, hard to extend, difficult to share template configuration between sub-charts and have security concerns.

Helm is may be more useful if we want to publish a chart with abstracted complexe cluster deployment configuration but for the reana-cluster use case I found that Kustomize was better suited, easiest and more "the Kubernetes way".

My current really WIP implementation for the moment: https://github.com/Shikanime/reana-cluster/tree/refactor/kustomize.

[diegodelemos]

Hello @Shikanime!

Great to see progress and a nice analysis of the tools 🙂.

Since there are more people interested in this change in the community (for example @BenGalewsky, @lukasheinrich, @khurtado, and @reanahub/developers in general), it would be great if you could share some more examples of the specific hiccups you found while using Helm for the reana-cluster use case, so we can all better understand which decision to take and more importantly, why we take it :)

Also, in order to help to clarify, I see in your branch that you have nice history, using Helm and now Kustomize, maybe splitting the history into two branches (Helm and Kustomize) and having a short usage scenario for each would also help to make a decision?

P.S. We should also take a closer look at @BenGalewsky first attempt with Helm which I believe can be found here https://github.com/BenGalewsky/reana-cluster/tree/helm_chart and maybe getting some more feedback?

[lukasheinrich]

While I think there are legitimate concerns with helm, it is I very broadly deployed compared to kustomize. I would vote for preparing a helm chart.

On Tue, Oct 22, 2019 at 1:45 PM Diego [email protected] wrote:

Hello @Shikanime https://github.com/Shikanime!

Great to see progress and a nice analysis of the tools 🙂.

Since there are more people interested in this change in the community (for example @BenGalewsky https://github.com/BenGalewsky, @lukasheinrich https://github.com/lukasheinrich, @khurtado https://github.com/khurtado, and @reanahub/developers https://github.com/orgs/reanahub/teams/developers in general), it would be great if you could share some more examples of the specific hiccups you found while using Helm for the reana-cluster use case, so we can all better understand which decision to take and more importantly, why we take it :)

Also, in order to help to clarify, I see in your branch https://github.com/Shikanime/reana-cluster/commits/refactor/kustomize that you have nice history, using Helm and now Kustomize, maybe splitting the history into two branches (Helm and Kustomize) and having a short usage scenario for each would also help to make a decision?

P.S. We should also take a closer look at @BenGalewsky https://github.com/BenGalewsky first attempt with Helm which I believe can be found here https://github.com/BenGalewsky/reana-cluster/tree/helm_chart and maybe getting some more feedback?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/reanahub/reana-cluster/issues/219?email_source=notifications&email_token=AARV6AZ5PUPSAVUXGOZWBT3QP3RUPA5CNFSM4IK67FBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB5N3BI#issuecomment-544923013, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARV6A3PAFBBNU2Z364ZJS3QP3RUPANCNFSM4IK67FBA .

[shikanime]

Hello @everyone

Of course, to start with, I started the project using Helm since it is the tool I used the most. Since September, I have moved all my work to Kustomize. I will objectively state below the reason and how Kustomize solved my problems and why i now exclusively use Kustomize.

Helm is one of the first package managers for Kubernetes and is an industry standard for the moment after the vanilla Kubernetes YAML manifest. Like Ksonnet, Helm uses a programmatic model instead of a declarative model to produce Kubernetes manifests.

On the other hand, Kustomize is template-free and declarative to extend the manifests and is natively supported by Kubernetes.

Organization

Helm 2

Helm uses the Chart.yaml file as an entry point that defines the version, name, description.... of the Chart. And values.yaml is where the parameters are defined to be passed down to the template.

./
├── Chart.yaml
├── values.yaml
├── requirements.lock
├── requirements.lock
├── files/
│   └── ...
├── charts/
│   ├── postgresql.tar.gz
│   ├── rabbitmq.tar.gz
│   ├── reana-ui
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   ├── requirements.lock
│   │   ├── requirements.lock
│   │   └── templates/
│   │       ├── *.tpl
│   │       ├── *.yaml
│   │       └── ...
│   └── ...
└── templates/
    ├── *.tpl
    ├── *.yaml
    └── ...

And it allows you to conditionally import nested subcharts and third-party charts.

./requirements.yaml:

dependencies:
  - name: postgreql
    version: 6.3.9
    repository: https://kubernetes-charts.storage.googleapis.com/
  - name: rabbitmq
    version: 6.7.4
    repository: https://kubernetes-charts.storage.googleapis.com/
  - name: reana-ui
    version: 0.1.0
    condition: reanaui.enabled
    repository: file://subcharts/reanaui

Kustomize

A Kustomize project structure is more minimalist and is unopinionated, using kustomization.yaml as the entry point for the package, structurally similar to Terraform. Kustomize allows us to model our structure according to our needs.

Environment oriented structure :

./
├── base/
│   ├── kustomization.yaml
│   │── *.yaml
│   └── ...
├── dev/
│   ├── base/
│   │   ├── kustomization.yaml
│   │   ├── *.yaml
│   │   └── ...
│   └── minikube/
│       ├── kustomization.yaml
│       ├── *.yaml
│       └── ...
└── prod/
    ├── base/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── google-cloud/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── cern/
        ├── kustomization.yaml
        ├── *.yaml
        └── ...

Overlays:

./
├── base/
│   ├── kustomization.yaml
│   │── *.yaml
│   └── ...
└── overlays/
    ├── cephfs/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    ├── minikube/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    ├── ui/
    │   ├── kustomization.yaml
    │   ├── *.yaml
    │   └── ...
    └── traefik/
        ├── kustomization.yaml
        ├── *.yaml
        └── ...

Kustomize in constrast is more declarative, kustomization.yaml hold every Kubernetes manifests in the resources list and other package imports in the bases using Hashicorp go getter format. And have several other utility such as:

• commonLabel to put a common every children resources;
• *Generator to create secret or configmap directly from litteral or from a file path.

./base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app.kubernetes.io/part-of: reana
  app.kubernetes.io/managed-by: kustomize
bases:
  - postgresql
  - redis
  - rabbitmq
  - reana-server
  - reana-ui
  - reana-wdb
  - reana-workflow-controller
# ...

./base/postgresql/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app.kubernetes.io/name: postgresql
  app.kubernetes.io/version: "11.5.0"
  app.kubernetes.io/component: database
resources:
  - headless-svc.yaml
  - statefulset.yaml
  - svc.yaml
secretGenerator:
  - literals:
      - postgresql-password=reana
    name: postgresql
# ...

Configuration

Helm

One of the main restrictions of templating is that it's not extensible by default. Historically, templating is good for ahead known configuration but in case of infrastructure deployment that's not the case, sometime we need to pass certain environment variable or volume based on various variables. Which lead to template a configuration to be configured on the Chart level.

./charts/reanaserver/templates/deployment.yaml

# ...
          env:
            {{- with .Values.hostName }}
            - name: REANA_URL
              value: {{ . }}
            {{- end }}
{{ include "reanaserver.env" . | indent 12 }}
{{- if .Values.extraEnv }}
{{ tpl (toYaml .Values.extraEnv) $ | indent 12 }}
{{- end }}
          ports:
            - name: rest-api
              containerPort: 5000
        - name: scheduler
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: ["flask", "start-scheduler"]
          resources:
{{- toYaml .Values.scheduler.resources | indent 12 }}
          {{- if .Values.securityContext.enabled }}
          securityContext:
            runAsUser: {{ .Values.securityContext.runAsUser }}
          {{- end }}
          env:
{{ include "reanaserver.env" . | indent 12 }}
{{- if .Values.extraEnv }}
{{ tpl (toYaml .Values.extraEnv) $ | indent 12 }}
{{- end }}
# ...

./values.yaml

reanaserver:
  extraVolumeMounts:
    - name: reana-shared-volume
      mountPath: "/var/reana"
  extraVolumes:
    - name: reana-shared-volume
      persistentVolumeClaim:
        claimName: manila-cephfs-pvc
        readOnly: false
  extraEnv:
    - name: SHARED_VOLUME_PATH
      value: reana-shared-volume
    - name: REANA_STORAGE_BACKEND
      value: "cephfs"

Kustomize

This is where Kustomize shine, it use template-free Kubernetes manifest and extend it by applying the superposition.

./base/postgresql/deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reana-server
spec:
  selector:
    matchLabels: &containerLabels
      app: reana-server
  template:
    metadata:
      labels: *containerLabels
    spec:
      containers:
        - env:
            - name: REANA_DEPLOYMENT_TYPE
              value: development
            - name: SHARED_VOLUME_PATH
              value: /var/reana
            - name: REANA_DB_HOST
              value: $(POSTGRESQL_NAME)
            - name: REANA_DB_PORT
              value: "5432"
            # ...
          name: server
          volumeMounts:
            - mountPath: /var/reana
              name: reana-shared-volume
          # ...

./prod/cern/postgresql/deploy-patch.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: reana-server
spec:
  template:
    spec:
      containers:
        - name: server
          env:
            - name: REANA_DEPLOYMENT_TYPE
              value: production
            - name: REANA_STORAGE_BACKEND
              value: "cephfs"
    - name: reana-shared-volume
      mountPath: "/var/reana"
  volumes:
    - name: reana-shared-volume
      persistentVolumeClaim:
        claimName: manila-cephfs-pvc
        readOnly: false

Deployment and security

Helm

Firstly Helm 2 manages its releases via the tiller, which is a security issue by itself in the same way as the RBAC security issues of the Kubenetes dashboard. Partially solved in the Helm 3 by using an opaque configmap instead of Tiller.

helm install --name reana-test ./chart

Kustomize

In contrast, Kustomize is natively supported by Kubernetes and is easier to deploy by simply using the following command on a directory that contains a file kustomization.yaml:

kubectl apply/create/delete -k ./manifests

Apply and prune all deleted resources:

kubectl apply --prune -k ./manifests

Conclution

At each step Kustomize feels more Kubernetes. With or without Kustomize, all the manifests written are valid vanilla Kubernetes manifest and are extensible by default without requiring to fork or create a pull request.

Helm is harsh and feels magical where Kustomize is declarative following the Kubernetes philosophy. The resources declared in Kustomize are ordered by priority such as CRD then configmap then... (https://github.com/helm/helm/issues/5871). Helm templates (*.tpl) are also another pain point point and cannot be shared or have strange behaviors between charts (https://github.com/helm/helm/issues/3920)