[TT-6206] Loading apps on tyk headless automatically on startup.

[nemmeviu]

Hi, guys

I'm using tyk version 3 with helm charts "tyk-helm/tyk-headless"

image:
  repository: docker.tyk.io/tyk-gateway/tyk-gateway
  tag: v3.2.1

We need to make "applications" and "policies" persistent when tyk-headless startup.

Looking inside values.yaml we found a "mounts" array, but we can't use this without making some changes.

Overall, we modified 3 files mentioned below:

• https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/secrets-mounts.yaml
• https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/deployment-gw-repset.yaml
• https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/values.yaml

With these changes, the policies file are loaded along with all the applications.

Is there any way to do this without our modifications? If not, we figured it could be useful to others to merge this upstream.

templates/secrets-mounts.yaml

Using the chart without changes, the files https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/secrets-mounts.yaml contains:

{{- range $secret := $deployment.mounts }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ $.Release.Name }}-{{ $key }}-secret-{{ $secret.name }}
  labels:
    app: {{ $key }}-secret-{{ $secret.name }}-{{ include "tyk-headless.fullname" $ }}
    chart: {{ include "tyk-headless.chart" $ }}
    release: {{ $.Release.Name }}
    heritage: {{ $.Release.Service }}
type: Opaque
data:
  {{ $secret.filename }}: |-
{{ $.Files.Get (printf "mounts/%s" $secret.filename) | b64enc | indent 4 }}
---
{{- end }}

To make possible to create a many applications, and merge this applications on tyk-gateway startup, we make this change:

{{- range $secret := $deployment.mounts }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ $.Release.Name }}-{{ $key }}-secret-{{ $secret.name }}
  labels:
    app: {{ $key }}-secret-{{ $secret.name }}-{{ include "tyk-headless.fullname" $ }}
    chart: {{ include "tyk-headless.chart" $ }}
    release: {{ $.Release.Name }}
    heritage: {{ $.Release.Service }}
type: Opaque
data:
  {{ $secret.filename }}: |-
{{ $.Files.Get (printf "mounts/%s/%s" $secret.subDir $secret.filename) | b64enc | indent 4 }}
---
{{- end }}

templates/deployments-gw-repset.yaml

Using the chart without changes, the files https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/deployments-gw-repset.yaml contains:

https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/deployment-gw-repset.yaml

line 154

           {{- if .Values.gateway.mounts }}
           {{- range $secret := .Values.gateway.mounts }}
           - name: {{ $.Release.Name }}-gateway-secret-{{ $secret.name }}
             mountPath: {{ $secret.mountPath }}
           {{- end }}
           {{- end }}

line 155

           {{- if .Values.gateway.mounts }}
           {{- range $secret := .Values.gateway.mounts }}
           - name: {{ $.Release.Name }}-gateway-secret-{{ $secret.name }}
             mountPath: {{ $secret.mountPath }}
             subPath: {{ $secret.filename }}
           {{- end }}
           {{- end }}

values.yaml

https://github.com/TykTechnologies/tyk-helm-chart/blob/master/tyk-headless/templates/deployment-gw-repset.yaml

  # Mount a subpath file
  mounts:
    - name: first-app
      filename: first-app.json
      mountPath: /mnt/tyk-gateway/apps/
      subDir: apps
    - name: second-app
      filename: second-app.json
      mountPath: /mnt/tyk-gateway/apps/
      subDir: apps
   - name: policies
     filename: policies.json
     mountPath: /mnt/tyk-gateway/policies/
     subDir: policies

Result

With this changes, we can restart all tyk-headless pods and the apps and policies will automatically loaded.

Exists other way to make the same?

[mhuaco]

Hello! Welcome! Thank you for reporting this, we will review internally and get back to you in the next few days.

[buraksekili]

Hi @nemmeviu, thank you for raising this and proposing possible solutions to the problem. 

I would like to get your reasoning to understand your problem deeply. 

As far as I understand, you want to make the apps and policies folder persistent. Since they are volume mounted using mountPath, they are overwritten by each startup - which is not what you want to achieve for persistency. Did I understand your problem correctly?

At first glance, this behavior looks normal to me, since recreating pods will create new mounts each time. Did you try using something like PVC?

It would be great if you provide some details about your use case to help me to understand the issue here. I am happy to help! In the meantime, I will investigate this issue and hope to come back to you with possible solutions.

[nemmeviu]

Hi @buraksekili, thank you for considering this issue. Give me a few more weeks for prepare and make this tests. Sorry for the delay.

[cuttingedge1109]

I think this https://github.com/TykTechnologies/tyk-helm-chart/pull/243 can be a solution for your usecase?

[totogo]

@buraksekili This is causing another issue for us in Kubernetes. When the gateway pod restarts somehow, all the API definitions are gone. We have to manually redeploy all the APIs with tyk-operator

As far as I understand, you want to make the apps and policies folder persistent. Since they are volume mounted using mountPath, they are overwritten by each startup - which is not what you want to achieve for persistency. Did I understand your problem correctly?

@cuttingedge1109 's PR #243 could fix the issue.

[buraksekili]

@totogo thanks for raising your issue. I guess we can continue with @cuttingedge1109's PR.

@cuttingedge1109 can you please update your PR based on the review? So that we can continue with your contribution.

if you need any help, I am more than happy to help.

[cuttingedge1109]

@totogo thanks for raising your issue. I guess we can continue with @cuttingedge1109's PR.

@cuttingedge1109 can you please update your PR based on the review? So that we can continue with your contribution.

if you need any help, I am more than happy to help.

@buraksekili I updated the PR and waiting review.

[buraksekili]

thank you @cuttingedge1109, will review it today or next week.

[caroltyk]

Thank you all for the contributions! We have added support for persistent volume mount in v0.14.0 through this PR https://github.com/TykTechnologies/tyk-helm-chart/pull/264.

There is a demo of how to use the latest helm chart + PVC to synchronise multiple gateways too: https://github.com/buraksekili/multiple-tyk-gateway

If you're an Open Source user, please take a look at our new Helm Charts which will be continued enhanced with latest feature: https://github.com/TykTechnologies/tyk-charts/tree/main/tyk-oss