golang package are not cp to the .gitlab-ci-local/cache directory

[BLaurent]

I have a job that build a golang binaries. After downloading all the dependencies the job failed to cp all dependencies to the cache directory

Minimal .gitlab-ci.yml illustrating the issue

---
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    make build

Expected behavior I was expected for the cache mechanism to work but instead I have :

Error: Command failed with exit code 1: docker cp ec31163541291737df9d215d3d79ca1d06d3603ec5dd411bd9a52170d2712b5b:/cache/. .gitlab-ci-local/cache/.
mkdir /Users/user/git/helm-plugin/.gitlab-ci-local/cache/default/.go/pkg/mod/github.com/!burnt!sushi/[email protected]/.github: permission denied
    at makeError (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Job.copyOut (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:1019:13)
    at /opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:935:21
    at Function.exclusive (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/mutex.ts:19:9)
    at Job.copyCacheOut (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:934:17)
    at Job.start (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:496:9)
    at /opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/p-map/index.js:57:22

Host information macos gitlab-ci-local 4.47.0

Containerd binary I am using docker

Additional context I've check the docker.io/firecow/gitlab-ci-local-util container and there is no volumes attached to it, despite this docker create

[firecow]

Are you aware that the commands are being executed on your local machine, and not inside a docker container ?

[BLaurent]

Hi @firecow , Thanks for taking the time to review this issue. I think my example was not correct:

variables:
  CI_DEBUG_TRACE: "true"
  FF_USE_FASTZIP: "true"
  GOLANG_VERSION: "1.22.1"
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
  cache:
    paths:
      - "${GOPATH}/pkg/mod/"
      - "${GOPATH}/bin"
image:
  name: golang:${GOLANG_VERSION}
stages:
  - build
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    go build

During this process two containers are launch golang:1.22.1 and gitlab-local-ci. At the end I end up with the same error, docker cp failed. So docker is involved somehow.

[firecow]

Hi @firecow , Thanks for taking the time to review this issue. I think my example was not correct:

variables:
  CI_DEBUG_TRACE: "true"
  FF_USE_FASTZIP: "true"
  GOLANG_VERSION: "1.22.1"
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
  cache:
    paths:
      - "${GOPATH}/pkg/mod/"
      - "${GOPATH}/bin"
image:
  name: golang:${GOLANG_VERSION}
stages:
  - build
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    go build

During this process two containers are launch golang:1.22.1 and gitlab-local-ci. At the end I end up with the same error, docker cp failed. So docker is involved somehow.

It cannot reproduce using this example.

[BLaurent]

I have setup this small test project to ensure we are seeing the same thing. Hope this will work for you Regards

[firecow]

Right, thanks that gave me/us the error to debug on

mjn@mjn-laptop:~/downloads/gitlab-ci-local-test$ gitlab-ci-local build-job 
Using fallback git user.email
parsing and downloads finished in 44 ms
build-job starting golang:1.22.1 (build)
build-job copied to docker volumes in 686 ms
build-job $ mkdir -p "${GOPATH}" # collapsed multi-line command
build-job $ go build # collapsed multi-line command
build-job > go: downloading github.com/labstack/echo/v4 v4.11.4
build-job > go: downloading golang.org/x/net v0.19.0
build-job > go: downloading github.com/labstack/gommon v0.4.2
build-job > go: downloading golang.org/x/crypto v0.17.0
build-job > go: downloading github.com/mattn/go-isatty v0.0.20
build-job > go: downloading github.com/mattn/go-colorable v0.1.13
build-job > go: downloading github.com/valyala/fasttemplate v1.2.2
build-job > go: downloading github.com/valyala/bytebufferpool v1.0.0
build-job > go: downloading golang.org/x/sys v0.15.0
build-job > go: downloading golang.org/x/text v0.14.0
build-job finished in 11 s
build-job > still running...
Error: Command failed with exit code 1: docker cp 3f78cb34d9b7be12b5b195ad2ead3ad391caa2ef86e240c7c924d83702120a93:/cache/. .gitlab-ci-local/cache/.
open /home/mjn/downloads/gitlab-ci-local-test/.gitlab-ci-local/cache/default/.go/pkg/mod/github.com/labstack/echo/[email protected]/.editorconfig: permission denied
    at makeError (/snapshot/firecow-gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/snapshot/firecow-gitlab-ci-local/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Job.copyOut (/snapshot/firecow-gitlab-ci-local/src/job.ts:1019:13)
    at /snapshot/firecow-gitlab-ci-local/src/job.ts:935:21
    at Function.exclusive (/snapshot/firecow-gitlab-ci-local/src/mutex.ts:19:9)
    at Job.copyCacheOut (/snapshot/firecow-gitlab-ci-local/src/job.ts:934:17)
    at Job.start (/snapshot/firecow-gitlab-ci-local/src/job.ts:496:9)
    at /snapshot/firecow-gitlab-ci-local/node_modules/p-map/index.js:57:22

[ANGkeith]

Under the hood, we're using docker cp to copy out the cache and it has the following "limitation":

docker cp creates directory without write permission set. Then fails to write files into it. https://stackoverflow.com/a/45276559/11054476

Unluckily, in golang, the packages installed are read-only resulting in this bug

For now, i'll recommend the following tempfix:

  script:
    - go build
    - |
      # HACK: https://github.com/firecow/gitlab-ci-local/issues/1170#issuecomment-2081111816
      if [[ $GITLAB_CI == "false" ]]; then
        find "${GOPATH}/pkg/mod/" -type d -exec chmod +w {} +
      fi

[firecow]

I wonder what Gitlab CI/CD runners use...

[pklapperich]

@firecow So I just checked what's in the cache on our deployed Gitlab runner and each cache is a zip file (not even a tar.gz, but a zip file).

The cache folder on the server is organized like:

/{GROUP}[/{subgroup}[/{subgroupN}...]]/{PROJECT}/{CACHE_KEY}/{CACHENAME}.zip

The contents of each zip is just all the files/folder structure with paths relative to the repo root.