GHA Cache intermittent - Ignoring `npm install` / node_modules

[zfalen-deloitte]

Contributing guidelines

• [X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• [X] The documentation does not mention anything about my problem
• [X] There are no open or closed issues that are related to my problem

Description

SO Link here: https://stackoverflow.com/questions/77623949/docker-caching-on-github-actions-npm-install-is-not-cached

I am using the documented examples for caching Docker build layers via the GitHub Actions cache described here.

It appears to be working mostly as intended. It does successfully create a cache - and seems to pull many layers from it - but it does not cache arguably the most important layer: npm-modules

No changes to the dependency files are occurring during/prior to the builds below. Theoretically, these layers should be 100% reusable unless we make changes to the package.json - and it would save my build time ~1.5min

My setup in the workflow:

      - name: Setup Docker `buildx`
        uses: docker/setup-buildx-action@v2

      - name: Build & Push Docker Image 🔨
        uses: docker/build-push-action@v4
        with:
          context: .
          file: Dockerfile.development
          push: true
          tags: |
            ${{ steps.docker-tags.outputs.tag }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Relevant stage of the Dockerfile:

# ------------------------------------------------------------------------------------------
# LAYER 2: Install dependencies only when needed
FROM base AS deps

# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat

# Create app directory
WORKDIR /usr/src/app

# Copy in dependency files
COPY .env.test .npmrc ./
COPY package.json package-lock.json ./

# Install production dependencies only
RUN npm ci --omit dev

Some steps are caching, from logs e.g.:

#8 [deps 1/5] RUN apk add --no-cache libc6-compat
#8 CACHED

#9 [deps 2/5] WORKDIR /usr/src/app
#9 CACHED

#10 [deps 3/5] COPY .env.test .npmrc ./
#10 CACHED

However, the next steps seemingly fail to cache - especially the npm ci step:

#11 [deps 4/5] COPY package.json package-lock.json ./
# .....blahblahblah
#11 sha256:bc7465dc4da3894941da9beb13e53fea672b4167e0031a049feea4cd6ed2cd79 40.11MB / 40.11MB 1.1s done
#11 DONE 2.2s

#....more regarding copying in the package.json....

#12 [deps 5/5] RUN npm ci --omit dev
#12 23.50 npm WARN deprecated @babel/[email protected]: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-properties instead.
#....blahblahblah
#12 75.77 added 1929 packages, and audited 1930 packages in 1m
#....blahblahblah
#12 DONE 76.9s

Weirdly, later in the Dockerfile I reference these modules:

# ------------------------------------------------------------------------------------------
# LAYER 3: Rebuild the source code only when needed
FROM node:18-alpine AS builder
WORKDIR /usr/src/app
COPY --from=deps /usr/src/app/node_modules ./node_modules

...which pulls from the cache, apparently.

#14 [builder 3/6] COPY --from=deps /usr/src/app/node_modules ./node_modules
#14 CACHED

Expected behaviour

GHA Cache should cache all layers which are unchanged between builds.

Actual behaviour

GHA Cache is caching only some layers, and not the node_modules - which is arguably the most important layer to cache.

IMPORTANT NOTE: Layer caching with this Dockerfile works as expected locally. That is to say, the npm ci step is cached/reused on my local machine using docker build when no changes to the dependencies are present.

YAML workflow

name: Trigger a Prerelease Via Push or Comment

on:
  pull_request:
    branches:
      - master
  issue_comment:
    types: [created]

env:
  IMAGE_NAME: some-image-name

jobs:
  prerelease:
    if: ${{ (github.event.issue.pull_request && contains(github.event.comment.body, '/prerelease')) || github.event.pull_request }} # check the comment if it contains the keywords or if its a pull request
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2

      - name: Checkout PR
        uses: dawidd6/action-checkout-pr@v1
        with:
          pr: ${{ github.event.pull_request.number || github.event.issue.number }}

      - name: Get Version From `package.json` 🏷️
        id: package-version
        uses: martinbeentjes/npm-get-version-action@master

      - name: Get List of Release Tags
        uses: octokit/[email protected]
        id: get-release-tags
        with:
          route: GET /repos/${{ github.repository_owner }}/${{ github.event.repository.name }}/git/refs/tags
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Bump Release Version
        uses: actions-ecosystem/action-bump-semver@v1
        id: bump-semver
        with:
          current_version: ${{ steps.package-version.outputs.current-version }}
          level: patch

      - name: Set Version as ENV
        id: pre-version
        run: |
          VERSION=${{ steps.bump-semver.outputs.new_version }}
          BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
          DOCKER_COMPLIANT_BRANCH_NAME=$(echo $BRANCH_NAME | sed 's/[^[:alnum:]-]/-/g' | tr '[:upper:]' '[:lower:]')

          PRE_VERSION="$VERSION-$DOCKER_COMPLIANT_BRANCH_NAME"
          echo PRE_VERSION=$PRE_VERSION

          echo "prerelease_full_version=$PRE_VERSION" >> $GITHUB_OUTPUT
          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT
          echo "docker_compliant_branch_name=$DOCKER_COMPLIANT_BRANCH_NAME" >> $GITHUB_OUTPUT
          echo "commit_ref=$(git rev-parse --verify HEAD)" >> $GITHUB_OUTPUT

      - name: Get Total Release Tags Matching This Version 🏷️
        id: total-tags
        run: |
          TOTAL_TAGS=$(echo '${{ steps.get-release-tags.outputs.data }}' | jq "[.[] | select(.ref | test(\"${{ steps.pre-version.outputs.prerelease_full_version }}\"; \"i\"))] | length")
          echo "value=$TOTAL_TAGS" >> $GITHUB_OUTPUT

      - name: Create Prerelease Tag 🏷️ #e.g. 2.0.58-enhancement-better-docker-caching.5
        id: prerelease-tag
        run: |
          PREVIOUS_PRERELEASE_VERSION_NAME=$(echo "${{ steps.pre-version.outputs.prerelease_full_version }}.${{ steps.total-tags.outputs.value }}")
        
          PRERELEASE_VERSION_NUMBER=$((${{ steps.total-tags.outputs.value }} + 1))
          PRERELEASE_VERSION_NAME=$(echo "${{ steps.pre-version.outputs.prerelease_full_version }}.$PRERELEASE_VERSION_NUMBER")
          echo "previous_tag=$PREVIOUS_PRERELEASE_VERSION_NAME" >> $GITHUB_OUTPUT
          echo "tag=$PRERELEASE_VERSION_NAME" >> $GITHUB_OUTPUT
          echo "version=$PRERELEASE_VERSION_NUMBER" >> $GITHUB_OUTPUT

      - name: Create Docker Compliant Tags
        id: docker-tags
        run: |
          IMAGE_ID=ghcr.io/${{ github.repository_owner }}/$IMAGE_NAME

          # Change all uppercase to lowercase
          IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
          echo IMAGE_ID=$IMAGE_ID
          
          DOCKER_COMPLIANT_TAG=$(echo "$IMAGE_ID:${{ steps.prerelease-tag.outputs.tag }}")
          echo DOCKER_COMPLIANT_TAG=$DOCKER_COMPLIANT_TAG

          echo "tag=$DOCKER_COMPLIANT_TAG" >> $GITHUB_OUTPUT
          echo "IMAGE_FULL_TAG=$DOCKER_COMPLIANT_TAG" >> $GITHUB_ENV

      - name: Log into GitHub Container Registry 🔓
        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login https://ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Setup Docker `buildx`
        uses: docker/setup-buildx-action@v2

      - name: Build & Push Docker Image 🔨
        uses: docker/build-push-action@v4
        with:
          context: .
          file: Dockerfile.development
          push: true
          tags: |
            ${{ steps.docker-tags.outputs.tag }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Prerelease
        id: prerelease
        uses: softprops/action-gh-release@v1
        with:
          body: |
            Prerelease generated for ${{ steps.prerelease-tag.outputs.tag }}.

            Docker image available at `${{ steps.docker-tags.outputs.tag }}`

          prerelease: true
          target_commitish: ${{ steps.pre-version.outputs.commit_ref }}
          name: '${{ steps.prerelease-tag.outputs.tag }}'
          tag_name: '${{ steps.prerelease-tag.outputs.tag }}'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Comment PR
        uses: thollander/actions-comment-pull-request@v2
        with:
          message: 'Prerelease created!'
          pr_number: ${{ github.event.issue.number }}

[yukccy]

I am facing same issue that the npm ci step in Dockerfile has not being cached.

[lkraav]

I'm think also seeing (something like?) this, but surprisingly only for 1 app out of 6. Can't figure out what's different about it.

...
#8 [stage-1 1/3] FROM docker.io/library/nginx:1.19-alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2
#8 resolve docker.io/library/nginx:1.19-alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2 done
#8 DONE 0.0s

#9 [auth] ***/testers-portal:pull token for registry-1.docker.io
#9 DONE 0.0s

#10 importing cache manifest from ***/testers-portal:buildcache
#10 inferred cache manifest type: application/vnd.oci.image.index.v1+json done
#10 DONE 0.6s

#6 [internal] load build context
#6 transferring context: 2.17MB 0.1s done
#6 DONE 0.1s

#11 [client-app 2/7] WORKDIR /app
#11 CACHED

#12 [client-app 3/7] COPY [./package*.json, /app/]
#12 CACHED

#13 [client-app 4/7] RUN npm install --silent
#13 sha256:27bbc6afb145de99accc53bcb39c9c8f18fa87d6299209502377b68761bda221 0B / 104.79MB 0.2s
#13 sha256:3c461cc005eb2f5800f3de8399ddf0ceb9af4eb901febfc663213273f8c89a9f 164.68kB / 164.68kB 0.2s done
#13 sha256:cb1607bbeb42eb40ec9066fc85f1b7f8ef0d0d175ffcff70cfeb0fd4744828b6 99B / 99B 0.2s done
#13 sha256:98b00e0a6a079def65676f976e860e2067ac07536ceaecc04b5d19c64958a3c5 292B / 292B 0.2s done
#13 sha256:63ee7d0b743d2664350409e8297032641e454e77286e03edf996706abfd4553c 4.16kB / 4.16kB 0.1s done
...

[lguichard]

Confirmed, same problem on my CI.

[or-he-MA]

Same problem for me. Strangely when bypassing the build-push-action and simply running the cli command ( docker buildx build....) it's cacheing everything besides the node-modules, and when using the prebuilt action, it's happening from time to time.

[cloudcoke]

The same problem happens to me. Is there any solution?

FROM node:18.15-alpine AS base
RUN apk add tzdata && ln -snf /usr/share/zoneinfo/Asia/Seoul /etc/localtime

FROM base AS builder

WORKDIR /app

COPY package-lock.json package.json tsconfig.build.json tsconfig.json ./
RUN npm install

COPY src ./src
RUN npm run build

FROM base AS runner

WORKDIR /app

COPY --from=builder /app/package*.json /app/tsconfig*.json ./
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist

EXPOSE 3000

CMD [ "npm", "run", "start" ]

name: Docker Cache Test

on:
  push:
    branches: main

jobs:
  docker-cache-test:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set Up Buildx
        uses: docker/setup-buildx-action@v3
        with:
          platforms: linux/amd64

      - name: Build & Push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: '${{ steps.login-ecr.outputs.registry }}/actions_test:latest'
          cache-from: type=gha
          cache-to: type=gha, mode=max
          provenance: false

...
#10 [builder 2/5] COPY package-lock.json package.json tsconfig.build.json tsconfig.json ./
#10 DONE 1.2s

#11 [builder 3/5] RUN npm install
#11 10.91 
#11 10.91 added 714 packages, and audited 715 packages in 10s
#11 10.91 
#11 10.91 116 packages are looking for funding
#11 10.91   run `npm fund` for details
#11 10.92 
#11 10.92 found 0 vulnerabilities
#11 10.92 npm notice 
#11 10.92 npm notice New major version of npm available! 9.5.0 -> 10.5.2
#11 10.92 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.5.2>
#11 10.92 npm notice Run `npm install -g [email protected]` to update!
#11 10.92 npm notice 
#11 DONE 11.0s

#12 [builder 4/5] COPY src ./src
#12 DONE 1.1s

#13 [builder 5/5] RUN npm run build
#13 0.536 
#13 0.536 > [email protected] build
#13 0.536 > nest build
#13 0.536 
#13 DONE 4.8s

#14 [runner 2/4] COPY --from=builder /app/package*.json /app/tsconfig*.json ./
#14 CACHED

#15 [runner 3/4] COPY --from=builder /app/node_modules ./node_modules
#15 CACHED

#15 [runner 3/4] COPY --from=builder /app/node_modules ./node_modules
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 0B / 31.17MB 0.2s
#15 sha256:546c24b708c7d128e445c9a0461c7072e0d3d0c0abbaea7abc4e9d8f760303f1 81.60kB / 81.60kB 0.2s done
#15 extracting sha256:546c24b708c7d128e445c9a0461c7072e0d3d0c0abbaea7abc4e9d8f760303f1 done
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 4.19MB / 31.17MB 0.5s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 6.29MB / 31.17MB 0.6s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 9.44MB / 31.17MB 0.8s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 13.63MB / 31.17MB 0.9s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 17.83MB / 31.17MB 1.1s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 22.02MB / 31.17MB 1.2s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 26.21MB / 31.17MB 1.4s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 30.41MB / 31.17MB 1.5s
#15 sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 31.17MB / 31.17MB 1.6s done
#15 extracting sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4
#15 extracting sha256:47368dd19a5d423d64f17607c08d29e7fc545064a8e1322a88937fbc7aeb74f4 3.3s done
#15 DONE 5.0s
...

[pkarolyi]

I am having the same issue, it also happens with docker compose build using the gha cache in a Github action. The layers are saved but then not used consistently even without changes to the repo. (I also tried having .git in .dockerignore but that didn't help)

[crazy-max]

Maybe cache is being evicted because you exceed storage limit: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

GitHub will remove any cache entries that have not been accessed in over 7 days. There is no limit on the number of caches you can store, but the total size of all caches in a repository is limited to 10 GB.

You can check that at https://github.com/<repo>/actions/caches

[zentavr]

Have the same behavior with the registry cache (using AWS ECR)