Pipeline Scale Optimization Questions

[caiocampoos]

Currently we are trying to migrate from CircleCI to tekton on a self hosted environment. The initial implementation was great but we are currently having issues while trying to scale our pipelines.

We have 2 monorepos with about 5 to 4k tests in each one using jest. A pipeline run with 20% to 30% of our tests runs about 5-6 minutes

When we run full tests we get something close to 20 minutes. I am currently after resources that helmp me optimize our operation, how to approach scaling issues.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: {{ .Values.projectName }}-pipeline
spec:
  workspaces:
    - name: shared-data
  params:
    - name: repo-url
      type: string
    - name: revision
      type: string
  tasks:
    - name: fetch-source
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-git-clone
          - name: namespace
            value: tekton-pipelines
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: main 
      workspaces:
        - name: output
          workspace: shared-data
    - name: install-deps
      runAfter: ["fetch-source"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-install-deps
          - name: namespace
            value: tekton-pipelines
      params: 
        - name: install-script
          value: {{ .Values.install_script }}
        - name: post-install-script
          value: {{ default "echo no script" .Values.post_install_script }}
      workspaces:
        - name: source
          workspace: shared-data
    {{- range $i, $value := .Values.packages }}
    - name: test-{{ $value }}
      runAfter: ["install-deps"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-test-pkg
          - name: namespace
            value: tekton-pipelines
      params:
        - name: pkg
          value: {{ $value }}
      workspaces:
        - name: source
          workspace: shared-data
    {{- end }}
    - name: list-source
      runAfter: ["test-{{ last .Values.packages }}"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-list-source
          - name: namespace
            value: tekton-pipelines
      workspaces:
        - name: source
          workspace: shared-data
   {{- range $serviceName, $service := .Values.services }}
    - name: deploy-{{ $service.name }} 
      runAfter: ["list-source"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-update-registry
          - name: namespace
            value: tekton-pipelines
      params:
        - name: DOCKERFILE
          value: {{ $service.DOCKERFILE }}
        - name: DOCKERFILE_CONTEXT_PATH
          value: {{ $service.DOCKERFILE_CONTEXT_PATH }}
        - name: IMAGE
          value: {{ $service.IMAGE }}
        - name: build_script
          value: {{ $service.build_script }}
        - name: image-tag
          value: $(tasks.fetch-source.results.commit)
      workspaces:
        - name: source
          workspace: shared-data
    - name: deploy-k8s-dev-{{ $service.name }} 
      runAfter: ["deploy-{{ $service.name }}"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: kubernetes-actions
          - name: namespace
            value: tekton-pipelines
      params:
        - name: "script"
          value: |
            export RELEASE_IMAGE=registry.woovi.dev/{{ $service.IMAGE }}:$(tasks.fetch-source.results.commit)
            kubectl get deployments -n {{ $service.target_namespace }}

            kubectl -n {{ $service.target_namespace }} set image deployment/{{ $service.deployment_name }} {{ $service.deployment_name }}=$RELEASE_IMAGE --record 
            kubectl  -n {{ $service.target_namespace }} rollout status deployment/{{ $service.deployment_name }} --timeout 5m
    {{- end }}

This is one of our pipelines, we currently have 6 services to build and deploy and about 25 packages from witch we run tests.

Some testing:

Two pipelines with ~50% load difference running each at a time, similar times: This is 10% of our tests

20-30% woovi server test packages:

The impact in performance is when running multiple pipelines at the same time:

I would appreciate any help if possible.

[caiocampoos]

Another issue i have encounter is to recreate our Jest cache, on CircleCI we are able to cache Jest, yarn and webpack in a persistent volume, there is a way to achieve a something similar with Tekton pipelines, can workspaces persist between pipelineRuns?

[zijiwork]

@caiocampoos Can you post a cache example?

[caiocampoos]

@caiocampoos Can you post a cache example?

https://dev.to/woovi/how-to-automate-tests-with-tekton-3caj

I wrote an article that shows an example with cache in the PipelineRun of a triggerTemplate. You have to create a pvc and declare as workspace.

From there you can pass to the pipelines and tasks. You can pass this workspace to the task as optional to make it easy.