Cache skipped because of Gitlab access token

[Tchekda]

Actual behavior Gitlab is proving a uniquely generated NPM_TOKEN to the docker context in order to download private dependencies.

Because of that the step ARG NPM_TOKEN isn't cached since this token changes each time. And so the build redownloads the whole node_modules folder which is fully unnecessary if packages.json and yarn.lock didn't not change.

I hoped that kaniko cache could do something about it but since the hash is different, the cached layer isn't found.

Expected behavior Skip dependencies re-download if packages.json and yarn.lock didn't change

Additional Information

1. Pipeline config

.compile_typescript:
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    IMAGE_TAG: ${CI_REGISTRY_IMAGE}/${PACKAGE_NAME}:${CI_COMMIT_SHORT_SHA}
  tags: [linux]
  stage: build
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "develop"
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
  script:
    - echo "Running PRE_BUILD_CMD"
    - eval $PRE_BUILD_CMD
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${DOCKERFILE_PATH}"
      --destination "${IMAGE_TAG}"
      --build-arg NPM_TOKEN=${CI_JOB_TOKEN}
      --cache=true

2. The docker file :

FROM node:16.14.2 AS development

ENV WORKDIR=/opt/api
WORKDIR ${WORKDIR}
ARG NPM_TOKEN

# Copy dependencies
COPY package.json yarn.lock ./

# Add NPM config for the private repository
RUN echo "@NAMESPACE:registry=https://GITLAB_PUBLIC_URL/api/v4/packages/npm/" > .npmrc \
    && echo "//GITLAB_PUBLIC_URL/api/v4/packages/npm/:_authToken=${NPM_TOKEN}">> .npmrc \
    && echo "//GITLAB_PUBLIC_URL/api/v4/projects/:_authToken=${NPM_TOKEN}">> .npmrc

# Install the dependences
RUN yarn config set "strict-ssl" false -g && yarn install --pure-lockfile --non-interactive 

# Copy code files 
COPY packages/api packages/api

# Build the code
RUN yarn api build

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [No]
Please check if the build works in docker but not in kaniko	- [No]
Please check if this error is seen when you use --cache flag	- [Yes]
Please check if your dockerfile is a multistage dockerfile	- [No]

[phongvq]

I think you can use personal access token or deploy token to workaround the issue, since these token will be same build after build.

https://docs.gitlab.com/ee/user/packages/npm_registry/#authenticate-with-a-personal-access-token-or-deploy-token

[aaron-prindle]

Can anyone here confirm if the Dockerfile cache would also invalidate in this case of dynamic ARG values (how this compares to docker's behaviour)?

[Tchekda]

I think you can use personal access token or deploy token to workaround the issue, since these token will be same build after build.

https://docs.gitlab.com/ee/user/packages/npm_registry/#authenticate-with-a-personal-access-token-or-deploy-token

Hello, Sorry I missed your reply. In our case it wouldn't be viable because we work as a team and having someone's PAT used in a pipeline isn't in line with our security measures. Also, if that person were to leave the team, all pipelines would break immediately

[Tchekda]

Can anyone here confirm if the Dockerfile cache would also invalidate in this case of dynamic ARG values (how this compares to docker's behaviour)?

Yes, since the hash of the layer wouldn't be the same, docker won't pull it from the cache.

[chris-ng-scmp]

Would be great if possible to escape selected ARG from the cache key...

[lucasmoreiradev]

Hey! Has anyone found a solution for this case?