Spike: analyze opportunities to shorten build times for main

[kylos101]

Is your feature request related to a problem? Please describe

The build for main appears to take between 3 and 5 hours to run. :grimacing:

In looking at the most recent successful build for main, it took 4h and 50m:

• dazzle build took 1h 28m
• dazzle combine took 2h 12m
• sync images with docker hub took 1h 8m

Describe the behaviour you'd like

A quicker loop, so we can iterate more easily.

Describe alternatives you've considered

For example, are we effectively using GAR cache with dazzle build and combine, or can that be improved? Is there a more efficient way to push content to Docker Hub...we spend a lot of time syncing, which checks tags that have already been pushed.

[Siddhant-K-code]

For example, are we effectively using GAR cache with dazzle build and combine, or can that be improved? Is there a more efficient way to push content to Docker Hub...we spend a lot of time syncing, which checks tags that have already been pushed.

I don't think, there is much possibility on Sync Images step. But, What would happen if we combine the Dazzle Build & Dazzle Combine steps as:

- name: 🔨 Dazzle build & 🖇️ Dazzle combine
  run: |
    dazzle build ${{ env.GAR_IMAGE_REGISTRY }}/gitpod-artifacts/docker-dev/workspace-base-images --chunked-without-hash
    dazzle build ${{ env.GAR_IMAGE_REGISTRY }}/gitpod-artifacts/docker-dev/workspace-base-images
    dazzle combine ${{ env.GAR_IMAGE_REGISTRY }}/gitpod-artifacts/docker-dev/workspace-base-images --all

Because, GH Actions doesn't caches b//w the steps. So, I think it would be good to build & combine the chunks in a single step. WDYT?

[Siddhant-K-code]

There is no much benefit from the above solution: saved ~8 minutes only 😞 logs here

[Siddhant-K-code]

I think we should include the following caching in our main workflow (to at least fasten up the push to main). We can Save & Restore Cache from main

      - name: 🗄️ Force Save Registry Cache of main branch as Per Sha
        uses: actions/cache@v3
        with:
          path: ~/registry
          key: ${{ runner.os }}-main-branch-cache-${{ github.sha }}

      - name: 🗄️ Restore Registry Cache of main branch
        uses: actions/cache@v3
        with:
          path: ~/registry
          key: ${{ runner.os }}-main-branch-cache-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-main-branch-cache-

Ref: .github/workflows/pull-request.yml#L38-L50

[kylos101]

From a Gitpod workspace, dazzle build and combine are a bit quicker (build is 30m, combine is 45m).

One option is to use our own runners, perhaps a Harvester VM. https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners#managing-access-to-your-runners

Will remove from the Team Workspace breakdown for now, and consult with Platform via their inbox.

[kylos101]

Hey @meysholdt , I added this to the Platform inbox to socialize using Harvester VMs to more quickly run the action associated with the build for our Main branch of workspace-images.

[meysholdt]

hey @kylos101, that's a great find. I've created this issue (https://github.com/gitpod-io/ops/issues/5484) to track us setting up a GitHub runner. I'm moving the original issue (the one that holds this comment) back to the workspace team's inbox since it seems to have a larger scope.