firecow
Compatibilty with running the Debian Salsa CI locally
This is halfway between a bug report and a feature request. Since the latest update to https://salsa.debian.org/salsa-ci-team/pipeline#running-the-pipeline-locally this gitlab-ci-local is now advertised as a potential tool for running the Salsa CI pipeline locally. Salsa CI is Debian's custom GitLab CI pipeline for testing Debian packaging git repositories before uploading Debian source packages to Debian archives.
I am however not able to fully run the pipeline locally, as the RELEASE variable does not propagate properly.
Steps to reproduce on a Debian/Ubuntu host:
# Install git-buildpackage
sudo apt install git-buildpackage --no-install-recommends
# Clone Debian packaging sources of an example package
gbp clone --verbose --add-upstream-vcs https://salsa.debian.org/debian/entr.git
cd entr
# Preview seems to work correctly
gitlab-ci-local --file debian/salsa-ci.yml --preview
# The list seems mostly correct, but some test jobs are missing
gitlab-ci-local --file debian/salsa-ci.yml --list
name description stage when allow_failure needs
extract-source provisioning on_success false
build build on_success false
build armel build on_success false
build armhf build on_success false
build arm64 build on_success false
build riscv64 build on_success false
build source build on_success false
upgrade from bookworm upgrade package and distro on_success false
upgrade from bullseye upgrade package and distro on_success false
upgrade from buster upgrade package and distro on_success false
upgrade from stretch upgrade package and distro on_success false
# Run the entire pipeline locally
gitlab-ci-local --file debian/salsa-ci.yml
The first job extract-sources passes correctly, but the second job build fails on:
parsing and downloads finished in 318 ms.
json schema validated in 133 ms
build starting registry.salsa.debian.org/salsa-ci-team/pipeline/base: (build)
Error: Command failed with exit code 125: docker pull registry.salsa.debian.org/salsa-ci-team/pipeline/base:
Error: invalid reference format
at makeError (/snapshot/firecow-gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
at <anonymous> (/snapshot/firecow-gitlab-ci-local/node_modules/execa/index.js:118:26)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async i (/snapshot/firecow-gitlab-ci-local/src/job.ts:1003:13)
at async Ut.pullImage (/snapshot/firecow-gitlab-ci-local/src/job.ts:1016:13)
at async Ut.start (/snapshot/firecow-gitlab-ci-local/src/job.ts:531:13)
at async <anonymous> (/snapshot/firecow-gitlab-ci-local/node_modules/p-map/index.js:57:22)
This error is from Podman/Docker as the image is missing the tag. I can get past this by defining the RELEASE manually:
gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable build
However, this command will complain about /builds and /gcl-builds not existing and fail:
build $ chmod o-w /builds
build > chmod: cannot access '/builds': No such file or directory
build finished in 14 s FAIL 1
build Running after script...
With the steps above to repeat this, could you please advice if you have any tips to make Salsa CI more compatible with gitlab-ci-local? Or is there something in gitlab-ci-local that could be improved to be able to run Salsa CI as-is?
Thanks for maintaining this tool!
@ottok Also add a link to a gitlab.com repository, where we can see your work in action. It makes it a lot easier to debug
The steps above have all info. Open with browser https://salsa.debian.org/debian/entr to browse pipelines in action.
On Fri., Jan. 31, 2025, 11:49 Mads Jon Nielsen, @.***> wrote:
@ottok https://github.com/ottok Also add a link to a gitlab.com repository, where we can see your work in action. It makes it a lot easier to debug
— Reply to this email directly, view it on GitHub https://github.com/firecow/gitlab-ci-local/issues/1473#issuecomment-2626911025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDINDAYZBIOWLHSIZPHOT2NNIMPAVCNFSM6AAAAABVRVOBGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWHEYTCMBSGU . You are receiving this because you were mentioned.Message ID: @.***>
The pipeline file is debian/salsa-ci.yml
On Fri., Jan. 31, 2025, 12:39 Otto Kekäläinen, @.***> wrote:
The steps above have all info. Open with browser https://salsa.debian.org/debian/entr to browse pipelines in action.
On Fri., Jan. 31, 2025, 11:49 Mads Jon Nielsen, @.***> wrote:
@ottok https://github.com/ottok Also add a link to a gitlab.com repository, where we can see your work in action. It makes it a lot easier to debug
— Reply to this email directly, view it on GitHub https://github.com/firecow/gitlab-ci-local/issues/1473#issuecomment-2626911025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDINDAYZBIOWLHSIZPHOT2NNIMPAVCNFSM6AAAAABVRVOBGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWHEYTCMBSGU . You are receiving this because you were mentioned.Message ID: @.***>
@ottok
However, this command will complain about /builds and /gcl-builds not existing and fail:
regarding ^,
https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/salsa-ci.yml?ref_type=heads#L433
update chmod o-w /builds to chmod o-w $CI_BUILDS_DIR
ps: currently CI_BUILDS_DIR is undefined in gcl
https://github.com/firecow/gitlab-ci-local/pull/1489
Thanks for spotting that, I filed now MR to fix it: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/577
I also tested that the build indeed proceeds past this point now. Next thing it fails on this:
$ git show
commit d7b92dea3bb3e41ac69c3572ac37f0956c831c97 (HEAD -> debian/latest)
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -1,5 +1,6 @@
include:
- - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
new file mode 100644
index 0000000..aef8eaf
--- /dev/null
+++ b/salsa.env
@@ -0,0 +1 @@
+RELEASE_FROM_CHANGELOG=unstable
$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable "build source"
...
build source > Updating symlinks in /usr/lib/ccache ...
build source $ PATH="/usr/lib/ccache/:${PATH}"
build source $ ccache -z
build source > Statistics zeroed
build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: missing operand after 'o-w'
build source > Try 'chmod --help' for more information.
build source finished in 24 s FAIL 1
build source Running after script...
build source imported cache 'build-amd64_' in 140 ms
build source imported artifacts in 78 ms
build source > time="2025-02-16T22:59:38Z" level=warning msg="The cgroupv2 manager is set to systemd but there is no systemd user session available"
build source > time="2025-02-16T22:59:38Z" level=warning msg="For using systemd, you may need to login using an user session"
build source > time="2025-02-16T22:59:38Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)"
build source > time="2025-02-16T22:59:38Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
build source $ CCACHE_DIR=${CCACHE_TMP_DIR} ccache -s
build source > Local storage:
build source > Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build source $ mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build source > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
build source finished in 25 s WARN 1 after_script
build source exported artifacts in 523 ms
build source copied artifacts to cwd in 91 ms
WARN build source after_script
FAIL build source
> Local storage:
> Cache size (GiB): 0.0 / 5.0 ( 0.00%)
> mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
Because of..
build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: missing operand after 'o-w'
..I also tried defining if manually:
$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/builds "build source"
...
build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: cannot access '/builds': No such file or directory
build source finished in 18 s FAIL 1
build source Running after script...
build source imported cache 'build-amd64_' in 137 ms
build source imported artifacts in 75 ms
build source > time="2025-02-16T23:04:22Z" level=warning msg="The cgroupv2 manager is set to systemd but there is no systemd user session available"
build source > time="2025-02-16T23:04:22Z" level=warning msg="For using systemd, you may need to login using an user session"
build source > time="2025-02-16T23:04:22Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)"
build source > time="2025-02-16T23:04:22Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
build source $ CCACHE_DIR=${CCACHE_TMP_DIR} ccache -s
build source > Local storage:
build source > Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build source $ mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build source > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
build source finished in 18 s WARN 1 after_script
build source exported artifacts in 519 ms
build source copied artifacts to cwd in 91 ms
WARN build source after_script
FAIL build source
> Local storage:
> Cache size (GiB): 0.0 / 5.0 ( 0.00%)
> mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
$ ls -la /builds/
drwxrwxrwx 2 admin admin 4096 Jan 20 21:46 .
drwxr-xr-x 7 admin root 4096 Jan 20 21:46 ..
I have at least one patch in Docker and I am a fairly experienced open source contributor. I might be able to debug this myself if you give some pointers on what the architecture/process here is, as I am unable to figure it our just by skimming your docs/code.
Thanks for maintaining gitlab-ci-local!
gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/builds "build source"
can you try to use --variable CI_BUILDS_DIR=/gcl-builds instead ? the CI_BUILDS_DIR is currently hard coded to /gcl-builds
I might be able to debug this myself if you give some pointers on what the architecture/process here is, as I am unable to figure it our just by skimming your docs/code.
hmm, not too sure, about this,
but probably trying to build from source should be a good starting point?
npm ci # install dependency
npm run dev # should run gitlab-ci-local with hot re-loading enabled
npm run test # for the regression test
Thanks, indeed, with gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds I get the builds to progress further now. In fact, when I removed the non-amd64 build jobs I got the whole pipeline to pass 🎉
$ git log -p
commit a74d6aa29f10ba8ae9a9801a9296a883c6aa905a (HEAD -> debian/latest)
Author: Otto <[email protected]>
Date: Tue Feb 18 02:45:45 2025 +0000
WIP: Disable non-amd64 builds
diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml
index 9e21e05..8bdb04f 100644
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -9,31 +9,6 @@ stages:
- upgrade package and distro
- publish # Stage referenced by Salsa CI template aptly stanza, so must exist even though not used
-# Additional builds on arm64 only when in main project as only it has runners
-# available in https://salsa.debian.org/groups/debian/-/runners
-# Don't use the SALSA_CI_DISABLE_BUILD_PACKAGE_ARM64 & co tags, as they don't
-# have this logic for testing group name and will end up making all pipelines
-# in forks stuck forever.
-build arm64:
- extends: .build-package-arm64
- rules:
- - if: $CI_PROJECT_ROOT_NAMESPACE == "debian"
-
-build armel:
- extends: .build-package-armel
- rules:
- - if: $CI_PROJECT_ROOT_NAMESPACE == "debian"
-
-build armhf:
- extends: .build-package-armhf
- rules:
- - if: $CI_PROJECT_ROOT_NAMESPACE == "debian"
-
-build riscv64:
- extends: .build-package-riscv64
- rules:
- - if: $CI_PROJECT_ROOT_NAMESPACE == "debian"
-
.test-prepare-container: &test-prepare-container |
# Change directory to ./debian/output where build artifacts can be found
cd ${WORKING_DIR}
commit e2b540858a0f70f93bbe1fe2b63eadf0decd000d
Author: Otto <[email protected]>
Date: Sun Feb 16 22:54:11 2025 +0000
WIP: Compatibility with gitlab-ci-local
diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml
index 865f4f5..9e21e05 100644
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -1,5 +1,6 @@
include:
- - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds
...
PASS extract-source
PASS build
PASS build source
PASS upgrade from bookworm
PASS upgrade from bullseye
PASS upgrade from buster
Another example:
gbp clone --verbose https://salsa.debian.org/debian/trend.git
cd trend
nano debian/salsa-ci.yml
include:
- - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
+ - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds
...
PASS extract-source
PASS build
PASS build source
pipeline finished in 43 s
Once https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/577 is merged any typical debian/salsa-ci.yml will run out-of-the-box.
Once #1489 is merged I assume there will be no more need to pass --variable CI_BUILDS_DIR=/gcl-builds to the command.
Next thing I am trying to debug is why it only runs the 'provisioning' and 'build' stages, but not the 'test' stage. Compare the above jobs completed to full pipeline at https://salsa.debian.org/debian/trend/-/pipelines/817877
Next thing I am trying to debug is why it only runs the 'provisioning' and 'build' stages, but not the 'test' stage. Compare the above jobs completed to full pipeline at https://salsa.debian.org/debian/trend/-/pipelines/817877
yeah, lets revisit it, we've just merged a couple of fixes
As #1489 got merged last week, testing again:
$ dpkg -l gitlab-ci-local
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============-============-============-=============================================
ii gitlab-ci-local 4.58.0 amd64 Tired of pushing to test your .gitlab-ci.yml?
$ gbp clone --verbose https://salsa.debian.org/debian/trend.git
$ cd trend
$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --list
parsing and downloads finished in 409 ms.
json schema validated in 137 ms
name description stage when allow_failure needs
extract-source provisioning on_success false
build build on_success false
build source build on_success false
$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --list-all
parsing and downloads finished in 411 ms.
json schema validated in 136 ms
name description stage when allow_failure needs
extract-source provisioning on_success false
build build on_success false
build i386 build never false
build armel build never false
build armhf build never false
build arm64 build never false
build riscv64 build never false
build source build on_success false
aptly publish never false [build,build i386,build armel,build armhf,build arm64]
test-build-any test never false
test-build-all test never false
test-build-twice test never false
test-build-profiles test never false
test-crossbuild-arm64 test never false
reprotest test never false [build]
lintian test never false [build]
autopkgtest test never false [build]
blhc test never false [build]
piuparts test never false [build]
missing-breaks test never true [build]
rc-bugs test never false [build]
wrap-and-sort test never false [extract-source]
atomic-reprotest: [aslr] test never false [build]
atomic-reprotest: [build_path] test never false [build]
atomic-reprotest: [domain_host] test never false [build]
atomic-reprotest: [environment] test never false [build]
atomic-reprotest: [exec_path] test never false [build]
atomic-reprotest: [fileordering] test never false [build]
atomic-reprotest: [home] test never false [build]
atomic-reprotest: [kernel] test never false [build]
atomic-reprotest: [locales] test never false [build]
atomic-reprotest: [num_cpus] test never false [build]
atomic-reprotest: [time] test never false [build]
atomic-reprotest: [timezone] test never false [build]
atomic-reprotest: [umask] test never false [build]
atomic-reprotest: [user_group] test never false [build]
$ gitlab-ci-local --file debian/salsa-ci.yml
...
build > removed directory '/builds/debian/trend/../.trend.ccache/9/6'
build > removed '/builds/debian/trend/../.trend.ccache/9/c/stats'
build > removed directory '/builds/debian/trend/../.trend.ccache/9/c'
build > removed '/builds/debian/trend/../.trend.ccache/9/9/stats'
build > removed directory '/builds/debian/trend/../.trend.ccache/9/9'
build > removed directory '/builds/debian/trend/../.trend.ccache/9'
build > removed directory '/builds/debian/trend/../.trend.ccache'
build $ cd ${WORKING_DIR}
build $ rm -rf ${WORKING_DIR}/${SOURCE_DIR}
build $ du -sh
build > 112K .
build $ if [ "$(du -s | cut -f1)" -gt ${SALSA_CI_MAX_ARTIFACTS_SIZE} ] # collapsed multi-line command
build finished in 26 s
build exported cache .ccache 'build-amd64_' in 1.37 s
build exported artifacts in 515 ms
build copied artifacts to cwd in 91 ms
PASS extract-source
PASS build
PASS build source
pipeline finished in 46 s
Based on my testing it seems that one does not even need to use --variable RELEASE=unstable anymore. This is looking great! Thanks! 🙏 🙏 🙏
Moving ccache dir emits error
When running rebuilds, I can see that the local ccache is used, and hit rate is 100%, but there is an error message mv: inter-device move failed: '/builds/debian/trend/../.trend.ccache' to '/builds/debian/trend/.ccache/.trend.ccache'; unable to remove target: Directory not empty to which I am unable to figure out the root cause.
build $ if [ -x /usr/bin/ccache ]; then # collapsed multi-line command
build > Cache directory: /builds/debian/trend/../.trend.ccache
build > Config file: /builds/debian/trend/../.trend.ccache/ccache.conf
build > System config file: /etc/ccache.conf
build > Stats updated: Sat Mar 8 19:42:57 2025
build > Stats zeroed: Sat Mar 8 19:42:57 2025
build > Cacheable calls: 2 / 4 (50.00%)
build > Hits: 2 / 2 (100.0%)
build > Direct: 2 / 2 (100.0%)
build > Preprocessed: 0 / 2 ( 0.00%)
build > Misses: 0 / 2 ( 0.00%)
build > Uncacheable calls: 2 / 4 (50.00%)
build > Autoconf compile/link: 0 / 2 ( 0.00%)
build > Bad compiler arguments: 0 / 2 ( 0.00%)
build > Called for linking: 1 / 2 (50.00%)
build > Called for preprocessing: 0 / 2 ( 0.00%)
build > Ccache disabled: 0 / 2 ( 0.00%)
build > Compilation failed: 0 / 2 ( 0.00%)
build > Compiler output file missing: 0 / 2 ( 0.00%)
build > Compiler produced empty output: 0 / 2 ( 0.00%)
build > Compiler produced stdout: 0 / 2 ( 0.00%)
build > Could not use modules: 0 / 2 ( 0.00%)
build > Could not use precompiled header: 0 / 2 ( 0.00%)
build > Forced recache: 0 / 2 ( 0.00%)
build > Multiple source files: 0 / 2 ( 0.00%)
build > No input file: 1 / 2 (50.00%)
build > Output to stdout: 0 / 2 ( 0.00%)
build > Preprocessing failed: 0 / 2 ( 0.00%)
build > Unsupported code directive: 0 / 2 ( 0.00%)
build > Unsupported compiler option: 0 / 2 ( 0.00%)
build > Unsupported environment variable: 0 / 2 ( 0.00%)
build > Unsupported source language: 0 / 2 ( 0.00%)
build > Errors: 0 / 4 ( 0.00%)
build > Compiler check failed: 0
build > Could not find compiler: 0
build > Could not read or parse input file: 0
build > Could not write to output file: 0
build > Error hashing extra file: 0
build > Input file modified during compilation: 0
build > Internal error: 0
build > Missing cache file: 0
build > Successful lookups:
build > Direct: 2 / 2 (100.0%)
build > Preprocessed: 0
build > Local storage:
build > Cache size (GiB): 0.0 / 5.0 ( 0.01%)
build > Files: 4
build > Cleanups: 0
build > Hits: 2 / 2 (100.0%)
build > Misses: 0 / 2 ( 0.00%)
build > Reads: 4
build > Writes: 0
build > Remote storage:
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Errors: 0
build > Timeouts: 0
build > mv: inter-device move failed: '/builds/debian/trend/../.trend.ccache' to '/builds/debian/trend/.ccache/.trend.ccache'; unable to remove target: Directory not empty
build finished in 24 s WARN 1 after_script
build finished in 24 s
build exported cache .ccache 'build-amd64_' in 571 ms
build exported artifacts in 527 ms
build copied artifacts to cwd in 95 ms
PASS extract-source
PASS build
PASS build source
WARN build after_script
WARN build source after_script
pipeline finished in 35 s
This is what the filesystem has:
$ sudo find / -name .trend.ccache
/home/admin/trend/.gitlab-ci-local/cache/build-amd64_/.ccache/.trend.ccache
/containers/storage/vfs/dir/d3fcb094302b72a1d9e275d21c7feb931b8475d2f4ee8a8ecf98f6851db51f58/cache/build-amd64_/.ccache/.trend.ccache
/containers/storage/vfs/dir/aeda4b4b7d00c802283918ceed04c6360bdbf4cfd88c62b220b07315f5fa079c/builds/debian/.trend.ccache
/containers/storage/vfs/dir/aeda4b4b7d00c802283918ceed04c6360bdbf4cfd88c62b220b07315f5fa079c/builds/debian/.trend.ccache/.trend.ccache
/containers/storage/vfs/dir/5b769e9ca3107cc4d3b6596504dd3dfa46250822f26d684bf95b1de64b424a5e/builds/debian/.trend.ccache
/containers/storage/vfs/dir/5b769e9ca3107cc4d3b6596504dd3dfa46250822f26d684bf95b1de64b424a5e/builds/debian/.trend.ccache/.trend.ccache
/containers/storage/vfs/dir/0b98dd648be2b845765faef0054aaffdc1f49f9e3a95a97ea5e85480a2f9009a/cache/build-amd64_/.ccache/.trend.ccache
/containers/storage/volumes/gcl-build-598625-build/_data/.ccache/.trend.ccache
/containers/storage/volumes/gcl-build-543462-build/_data/.ccache/.trend.ccache
/containers/storage/volumes/gcl-buildIAsource-671149-build/_data/.ccache/.trend.ccache
Stage 'test' not running
Another issue is that only the 'provisioning' and 'build' stages are run. I am unable to figure out how to run the other stages 'publish' and 'test':
$ gitlab-ci-local --file debian/salsa-ci.yml --list-all
WARN Avoid overriding predefined variables (https://gitlab.com/gitlab-org/gitlab/-/blob/v17.7.1-ee/doc/ci/variables/predefined_variables.md?plain=1&ref_type=tags#L15-16) [FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR] as it can cause the pipeline to behave unexpectedly.
parsing and downloads finished in 411 ms.
json schema validated in 138 ms
name description stage when allow_failure needs
extract-source provisioning on_success false
build build on_success false
build i386 build never false
build armel build never false
build armhf build never false
build arm64 build never false
build riscv64 build never false
build source build on_success false
aptly publish never false [build,build i386,build armel,build armhf,build arm64]
test-build-any test never false
test-build-all test never false
test-build-twice test never false
test-build-profiles test never false
test-crossbuild-arm64 test never false
reprotest test never false [build]
lintian test never false [build]
autopkgtest test never false [build]
...
$ gitlab-ci-local --file debian/salsa-ci.yml test
parsing and downloads finished in 406 ms.
json schema validated in 134 ms
test could not be found
On actual GitLab the 'test' does run: https://salsa.debian.org/debian/trend/-/pipelines/817877
Stage 'test' not running
Another issue is that only the 'provisioning' and 'build' stages are run. I am unable to figure out how to run the other stages 'publish' and 'test':
$ gitlab-ci-local --file debian/salsa-ci.yml --list-all WARN Avoid overriding predefined variables (https://gitlab.com/gitlab-org/gitlab/-/blob/v17.7.1-ee/doc/ci/variables/predefined_variables.md?plain=1&ref_type=tags#L15-16) [FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR] as it can cause the pipeline to behave unexpectedly. parsing and downloads finished in 411 ms. json schema validated in 138 ms name description stage when allow_failure needs extract-source provisioning on_success false build build on_success false build i386 build never false build armel build never false build armhf build never false build arm64 build never false build riscv64 build never false build source build on_success false aptly publish never false [build,build i386,build armel,build armhf,build arm64] test-build-any test never false test-build-all test never false test-build-twice test never false test-build-profiles test never false test-crossbuild-arm64 test never false reprotest test never false [build] lintian test never false [build] autopkgtest test never false [build] ... $ gitlab-ci-local --file debian/salsa-ci.yml test parsing and downloads finished in 406 ms. json schema validated in 134 ms test could not be foundOn actual GitLab the 'test' does run: https://salsa.debian.org/debian/trend/-/pipelines/817877
I'm running into the same issue mentioned above by @ottok, the test stage is not running at all in gitlab-ci-local, only provisioning and build run. On GitLab, the test stage runs fine. @firecow @ANGkeith, any idea what might be causing this difference?
@aquilamacedo @ottok
this is the output of --file debian/salsa-ci.yml --list-all after #1633
extract-source provisioning on_success false
build build on_success false
build i386 build on_success false
build arm64 build on_success false
build source build on_success false
test-build-any test on_success false [extract-source,build]
test-build-all test on_success false [extract-source,build]
reprotest test on_success false [build]
lintian test on_success false [build]
autopkgtest test on_success false [build]
autopkgtest i386 test on_success false [build]
blhc test on_success false [build]
piuparts test on_success false [build]
missing-breaks test on_success true [build]
upgrade from bookworm upgrade package and distro on_success false
upgrade from bullseye upgrade package and distro on_success false
upgrade from buster upgrade package and distro on_success false
upgrade from stretch upgrade package and distro on_success false
is there any jobs that's still missing?
In the examples above I have used packages trend and entr as examples. The pipeline of trend is a vanilla Salsa CI pipeline with no customizations:
---
include:
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
The pipeline of entr has a bunch of customizations and your "screenshot" above matches the list of jobs that entr currently has:
---
include:
- https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
...
build arm64:
...
build armel:
...
build armhf:
...
build riscv64:
...
upgrade from bookworm:
...
upgrade from bullseye:
...
upgrade from buster:
...
upgrade from stretch:
...
Hi.
I'm trying to run Salsa pipeline locally but without success. Source code is here: https://salsa.debian.org/maciej.lew/lifelines/-/tree/build-pipeline?ref_type=heads. That pipeline is Salsa pipeline without any customization.
When I run it in unprivileged mode I get ccache issue:
gitlab-ci-local --file debian/salsa-ci.yml build
build $ su salsa-ci -c "${MMDEBSTRAP_COMMAND}"
build > E: unable to pick chroot mode automatically (use --mode for manual selection)
build finished in 1.57 min FAIL 25
build Running after script...
build $ . ${CI_PROJECT_DIR}/ccache.env
build $ echo "ccache stats before building:"
build > ccache stats before building:
build $ cat ${CCACHE_STATS_BEFORE}
build $ echo "ccache stats after building:"
build > ccache stats after building:
build $ test ! "${ENABLE_CCACHE}" != "0" || CCACHE_DIR=${CCACHE_TMP_DIR} ccache -svv
build > Cache directory: /builds/maciej.lew/.lifelines.ccache
build > Config file: /builds/maciej.lew/.lifelines.ccache/ccache.conf
build > System config file: /etc/ccache.conf
build > Stats updated: never
build > Stats zeroed: never
build > Cacheable calls: 0
build > Hits: 0
build > Direct: 0
build > Preprocessed: 0
build > Misses: 0
build > Uncacheable calls: 0
build > Autoconf compile/link: 0
build > Bad compiler arguments: 0
build > Called for linking: 0
build > Called for preprocessing: 0
build > Ccache disabled: 0
build > Compilation failed: 0
build > Compiler output file missing: 0
build > Compiler produced empty output: 0
build > Compiler produced stdout: 0
build > Could not use modules: 0
build > Could not use precompiled header: 0
build > Forced recache: 0
build > Multiple source files: 0
build > No input file: 0
build > Output to stdout: 0
build > Preprocessing failed: 0
build > Unsupported code directive: 0
build > Unsupported compiler option: 0
build > Unsupported environment variable: 0
build > Unsupported source encoding: 0
build > Unsupported source language: 0
build > Errors: 0
build > Compiler check failed: 0
build > Could not find compiler: 0
build > Could not read or parse input file: 0
build > Could not write to output file: 0
build > Error hashing extra file: 0
build > Input file modified during compilation: 0
build > Internal error: 0
build > Missing cache file: 0
build > Local storage:
build > Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build > Files: 0
build > Cleanups: 0
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Remote storage:
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Errors: 0
build > Timeouts: 0
build $ test ! "${ENABLE_CCACHE}" != "0" || mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build > mv: cannot stat '/builds/maciej.lew/lifelines/../.lifelines.ccache': No such file or directory
build finished in 1.62 min WARN 1 after_script
WARNING: .ccache: no matching files. Ensure that the artifact path is relative to the working directory
build cache created in '.gitlab-ci-local/cache/build-amd64_' in 4.26 s
build exported artifacts in 3.49 s
build copied artifacts to cwd in 108 ms
WARN build after_script
FAIL build
> Errors: 0
> Timeouts: 0
> mv: cannot stat '/builds/maciej.lew/lifelines/../.lifelines.ccache': No such file or directory
When I run it in privileged mode, the ccache problem is solved, but downloading submodules fails:
build $ su salsa-ci -c "${MMDEBSTRAP_COMMAND}"
build > I: automatically chosen mode: unshare
build > I: chroot architecture amd64 is equal to the host's architecture
build > I: finding correct signed-by value...
build > I: automatically chosen format: tar
build > I: using /tmp/mmdebstrap.aPLAJdIzmP as tempdir
build > I: running apt-get update...
build > still running...
build > I: downloading packages with apt...
build > still running...
build > I: extracting archives...
build > W: skipping creation of ./dev/console because ./dev/console does not exist on the outside
build > I: installing essential packages...
build > still running...
build > still running...
build > I: installing remaining packages inside the chroot...
build > still running...
build > I: running --customize-hook in shell: sh -c 'chroot "$1" update-ccache-symlinks' exec /tmp/mmdebstrap.aPLAJdIzmP
build > I: cleaning package lists and apt cache...
build > I: creating tarball...
build > I: skipping output/dev as requested
build > still running...
build > I: done
build > I: removing tempdir /tmp/mmdebstrap.aPLAJdIzmP...
build > I: success in 668.0520 seconds
build $ mkdir -vp ${WORKING_DIR}
build > mkdir: created directory '/builds/maciej.lew/lifelines/debian/output'
build $ (set -o pipefail; rm -vrf ${CCACHE_TMP_DIR} | tail)
build $ if [ "${ENABLE_CCACHE}" = 1 ] ; # collapsed multi-line command
build > mkdir: created directory '/builds/maciej.lew/lifelines/.ccache'
build > created directory '/builds/maciej.lew/lifelines/../.lifelines.ccache'
build > removed directory '/builds/maciej.lew/lifelines/.ccache'
build $ mkdir -p ~salsa-ci/.config/sbuild/
build $ cat << "EOF" >> ~salsa-ci/.config/sbuild/config.pl # collapsed multi-line command
build $ mkdir -vp ${BUILD_DIR}
build $ if ! [ -z "${SALSA_CI_EXTRA_REPOSITORY_SOURCES}" ] ; then # collapsed multi-line command
build $ mkdir -vp ${WORKING_DIR} ${CCACHE_WORK_DIR}
build > mkdir: created directory '/builds/maciej.lew/lifelines/.ccache'
build $ if [ "${ENABLE_CCACHE}" = 1 ] ; # collapsed multi-line command
build $ if [ "${ENABLE_CCACHE}" = 1 ] ; # collapsed multi-line command
build $ if printf '%s\n' "${BUILD_TWICE}" "${SALSA_CI_VALIDATE_PACKAGE_CLEAN_UP}" | grep -qiE '^(1|yes|true)$' ; then # collapsed multi-line command
build $ test ! "${ENABLE_CCACHE}" != "0" || sed -i -e "s#%CCACHE_DIR%#${CCACHE_DIR}#" ~salsa-ci/.config/sbuild/config.pl
build $ chown -R salsa-ci ~salsa-ci/.config/sbuild/
build $ if [ "${ENABLE_CCACHE}" != "0" ]; then # collapsed multi-line command
build > CCACHE_STATS_BEFORE=/tmp/tmp.5UsOO9VolK
build $ chown -R 'salsa-ci:' ${WORKING_DIR} ${CCACHE_DIR}
build $ gbp pull --ignore-branch --pristine-tar --track-missing
build > gbp:info: Fetching from default remote for each branch
build > gbp:error: Error running git fetch: error: cannot run ssh: No such file or directory
build > fatal: unable to fork
build finished in 13 min FAIL 1
build Running after script...
build imported cache 'build-amd64_' in 1.39 s
build $ . ${CI_PROJECT_DIR}/ccache.env
build $ echo "ccache stats before building:"
build > ccache stats before building:
build $ cat ${CCACHE_STATS_BEFORE}
build > Cache directory: /builds/maciej.lew/.lifelines.ccache
build > Config file: /builds/maciej.lew/.lifelines.ccache/ccache.conf
build > System config file: /etc/ccache.conf
build > Stats updated: never
build > Stats zeroed: never
build > Cacheable calls: 0
build > Hits: 0
build > Direct: 0
build > Preprocessed: 0
build > Misses: 0
build > Uncacheable calls: 0
build > Autoconf compile/link: 0
build > Bad compiler arguments: 0
build > Called for linking: 0
build > Called for preprocessing: 0
build > Ccache disabled: 0
build > Compilation failed: 0
build > Compiler output file missing: 0
build > Compiler produced empty output: 0
build > Compiler produced stdout: 0
build > Could not use modules: 0
build > Could not use precompiled header: 0
build > Forced recache: 0
build > Multiple source files: 0
build > No input file: 0
build > Output to stdout: 0
build > Preprocessing failed: 0
build > Unsupported code directive: 0
build > Unsupported compiler option: 0
build > Unsupported environment variable: 0
build > Unsupported source encoding: 0
build > Unsupported source language: 0
build > Errors: 0
build > Compiler check failed: 0
build > Could not find compiler: 0
build > Could not read or parse input file: 0
build > Could not write to output file: 0
build > Error hashing extra file: 0
build > Input file modified during compilation: 0
build > Internal error: 0
build > Missing cache file: 0
build > Local storage:
build > Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build > Files: 0
build > Cleanups: 0
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Remote storage:
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Errors: 0
build > Timeouts: 0
build $ echo "ccache stats after building:"
build > ccache stats after building:
build $ test ! "${ENABLE_CCACHE}" != "0" || CCACHE_DIR=${CCACHE_TMP_DIR} ccache -svv
build > Cache directory: /builds/maciej.lew/.lifelines.ccache
build > Config file: /builds/maciej.lew/.lifelines.ccache/ccache.conf
build > System config file: /etc/ccache.conf
build > Stats updated: never
build > Stats zeroed: never
build > Cacheable calls: 0
build > Hits: 0
build > Direct: 0
build > Preprocessed: 0
build > Misses: 0
build > Uncacheable calls: 0
build > Autoconf compile/link: 0
build > Bad compiler arguments: 0
build > Called for linking: 0
build > Called for preprocessing: 0
build > Ccache disabled: 0
build > Compilation failed: 0
build > Compiler output file missing: 0
build > Compiler produced empty output: 0
build > Compiler produced stdout: 0
build > Could not use modules: 0
build > Could not use precompiled header: 0
build > Forced recache: 0
build > Multiple source files: 0
build > No input file: 0
build > Output to stdout: 0
build > Preprocessing failed: 0
build > Unsupported code directive: 0
build > Unsupported compiler option: 0
build > Unsupported environment variable: 0
build > Unsupported source encoding: 0
build > Unsupported source language: 0
build > Errors: 0
build > Compiler check failed: 0
build > Could not find compiler: 0
build > Could not read or parse input file: 0
build > Could not write to output file: 0
build > Error hashing extra file: 0
build > Input file modified during compilation: 0
build > Internal error: 0
build > Missing cache file: 0
build > Local storage:
build > Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build > Files: 0
build > Cleanups: 0
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Remote storage:
build > Hits: 0
build > Misses: 0
build > Reads: 0
build > Writes: 0
build > Errors: 0
build > Timeouts: 0
build $ test ! "${ENABLE_CCACHE}" != "0" || mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build > created directory '/builds/maciej.lew/lifelines/.ccache/.lifelines.ccache'
build > removed directory '/builds/maciej.lew/lifelines/../.lifelines.ccache'
build $ cd ${WORKING_DIR}
build $ du -sh
build > 4.0K .
build $ if [ "$(du -s | cut -f1)" -gt ${SALSA_CI_MAX_ARTIFACTS_SIZE} ] # collapsed multi-line command
.ccache: found 2 artifact files and directories
build cache created in '.gitlab-ci-local/cache/build-amd64_' in 4.56 s
build exported artifacts in 3.74 s
build copied artifacts to cwd in 728 ms
FAIL build
> created directory '/builds/maciej.lew/lifelines/.ccache/.lifelines.ccache'
> removed directory '/builds/maciej.lew/lifelines/../.lifelines.ccache'
> 4.0K .
Should I ran it privileged or unprivileged? How to solve ccache issue in unprivileged mode? How to pass submodule fetching (force http fetch)?
https://salsa.debian.org/maciej.lew/lifelines/-/commits/build-pipeline
I don't see a single pipeline run on any commits here. Please make sure Salsa CI itself works for the package in general before testing local builds or reporting issues in context of local builds.
Should I ran it privileged or unprivileged? How to solve ccache issue in unprivileged mode? How to pass submodule fetching (force http fetch)?
Since Salsa CI was refactored to use sbuild, the ccache has been broken in Salsa CI itself and won't work neither remotely or locally.
I suspect the other issues you are seeing are also related to having the pipeline refactored to use sbuild. Please contribute pipeline improvements or bug reports to https://salsa.debian.org/salsa-ci-team/pipeline about issues in Salsa CI itself.
I don't see a single pipeline run on any commits here. Please make sure Salsa CI itself works for the package in general before testing local builds or reporting issues in context of local builds.
It wasn't public, check now: https://salsa.debian.org/maciej.lew/lifelines/-/pipelines/960521 - it works in Salsa CI (at least build step).