wave
wave copied to clipboard
Published
20 hours ago •
seqeralabs
seqeralabs
Spack build failures
Getting a build failure using wave-cli to build the example spack image:
Details
java.lang.IllegalStateException: Unexpected response for 'https://wave.seqera.io/v2/wt/fbdd6fcaab96/wave/build/manifests/star-2.5.4a_bwa-0.7.15--1bbd3f1023331159': [400] {"errors":[{"code":"UNKNOWN","message":"INFO[0000] Resolved base name spack/ubuntu-jammy:v0.20.0 to builder \nINFO[0000] Retrieving image manifest spack/ubuntu-jammy:v0.20.0 \nINFO[0000] Retrieving image spack/ubuntu-jammy:v0.20.0 from registry index.docker.io \nINFO[0001] Retrieving image manifest spack/ubuntu-jammy:v0.20.0 \nINFO[0001] Returning cached image manifest \nINFO[0001] Retrieving image manifest ubuntu:22.04 \nINFO[0001] Retrieving image ubuntu:22.04 from registry index.docker.io \nINFO[0002] Retrieving image manifest ubuntu:22.04 \nINFO[0002] Returning cached image manifest \nINFO[0002] Built cross stage deps: map[0:[/opt/spack-env /opt/software /opt/._view]] \nINFO[0002] Retrieving image manifest spack/ubuntu-jammy:v0.20.0 \nINFO[0002] Returning cached image manifest \nINFO[0002] Retrieving image manifest spack/ubuntu-jammy:v0.20.0 \nINFO[0002] Returning cached image manifest \nINFO[0002] Executing 0 build triggers \nINFO[0002] Building stage 'spack/ubuntu-jammy:v0.20.0' [idx: '0', base-idx: '-1'] \nINFO[0002] Checking for cached layer 195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build/cache:3c7f60a8c5305502134b52bf40f93ae4046b83466c7748fee70f1cd0015fb175... \nINFO[0002] Using caching version of cmd: RUN mkdir -p /opt/spack-env && sed -i -e 's;compilers:;compilers::;' -e 's;^ *flags: *{}; flags:\\n cflags: -O3\\n cxxflags: -O3\\n fflags: -O3;' /root/.spack/linux/compilers.yaml && cd /opt/spack-env && spack config add config:install_tree:/opt/software && spack config add concretizer:unify:true && spack config add concretizer:reuse:true && spack config add packages:all:target:[x86_64] && printf \" view: /opt/view\\n\" >> /opt/spack-env/spack.yaml \nINFO[0002] Checking for cached layer 195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build/cache:56d87e8adae6f90946598d324a4c75098ad89484ce3dbe716f793de57e15f07f... \nINFO[0003] Using caching version of cmd: RUN cd /opt/spack-env && fingerprint=\"$(spack gpg trust /var/seqera/spack/key 2>&1 | tee /dev/stderr | sed -nr \"s/^gpg: key ([0-9A-F]{16}): secret key imported$/\\1/p\")\" && spack mirror add seqera-spack s3://seqera-production-spack-binarycache && spack mirror add binary_mirror https://binaries.spack.io/releases/v0.20 && spack buildcache keys --install --trust && spack -e . concretize -f && spack --env . install && spack -e . buildcache push --allow-root --key \"$fingerprint\" s3://seqera-production-spack-binarycache && spack gc -y && ( find -L /opt/._view/* -type f -exec readlink -f '{}' \\; | xargs file -i | grep 'charset=binary' | grep 'x-executable\\|x-archive\\|x-sharedlib' | awk -F: '{print $1}' | xargs strip -s ) || true \nINFO[0003] Checking for cached layer 195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build/cache:b4aa60d3c34f8ca831279ec0cf8ebd7dc1bcca4e387cb61757fcebd37021d784... \nINFO[0003] No cached layer found for cmd RUN cd /opt/spack-env && spack env activate --sh -d . >> /opt/spack-env/z10_spack_environment.sh && original_view=$( cd /opt ; ls -1d ._view/* ) && sed -i \"s;/view/;/$original_view/;\" /opt/spack-env/z10_spack_environment.sh && echo \"# Needed for Perl applications\" >>/opt/spack-env/z10_spack_environment.sh && echo \"export PERL5LIB=$(eval ls -d /opt/._view/*/lib/5.*):$PERL5LIB\" >>/opt/spack-env/z10_spack_environment.sh && rm -rf /opt/view \nINFO[0003] Unpacking rootfs as cmd COPY spack.yaml /opt/spack-env/spack.yaml requires it. \nINFO[0023] COPY spack.yaml /opt/spack-env/spack.yaml \nINFO[0023] Taking snapshot of files... \nINFO[0023] ARG AWS_STS_REGIONAL_ENDPOINTS \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] ARG AWS_REGION \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] ARG AWS_DEFAULT_REGION \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] ARG AWS_ROLE_ARN \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] ARG AWS_WEB_IDENTITY_TOKEN_FILE \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] ENV AWS_STS_REGIONAL_ENDPOINTS=${AWS_STS_REGIONAL_ENDPOINTS} AWS_REGION=${AWS_REGION} AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} AWS_ROLE_ARN=${AWS_ROLE_ARN} AWS_WEB_IDENTITY_TOKEN_FILE=${AWS_WEB_IDENTITY_TOKEN_FILE} \nINFO[0023] No files changed in this command, skipping snapshotting. \nINFO[0023] RUN mkdir -p /opt/spack-env && sed -i -e 's;compilers:;compilers::;' -e 's;^ *flags: *{}; flags:\\n cflags: -O3\\n cxxflags: -O3\\n fflags: -O3;' /root/.spack/linux/compilers.yaml && cd /opt/spack-env && spack config add config:install_tree:/opt/software && spack config add concretizer:unify:true && spack config add concretizer:reuse:true && spack config add packages:all:target:[x86_64] && printf \" view: /opt/view\\n\" >> /opt/spack-env/spack.yaml \nINFO[0023] Found cached layer, extracting to filesystem \nINFO[0024] RUN cd /opt/spack-env && fingerprint=\"$(spack gpg trust /var/seqera/spack/key 2>&1 | tee /dev/stderr | sed -nr \"s/^gpg: key ([0-9A-F]{16}): secret key imported$/\\1/p\")\" && spack mirror add seqera-spack s3://seqera-production-spack-binarycache && spack mirror add binary_mirror https://binaries.spack.io/releases/v0.20 && spack buildcache keys --install --trust && spack -e . concretize -f && spack --env . install && spack -e . buildcache push --allow-root --key \"$fingerprint\" s3://seqera-production-spack-binarycache && spack gc -y && ( find -L /opt/._view/* -type f -exec readlink -f '{}' \\; | xargs file -i | grep 'charset=binary' | grep 'x-executable\\|x-archive\\|x-sharedlib' | awk -F: '{print $1}' | xargs strip -s ) || true \nINFO[0024] Found cached layer, extracting to filesystem \nINFO[0025] RUN cd /opt/spack-env && spack env activate --sh -d . >> /opt/spack-env/z10_spack_environment.sh && original_view=$( cd /opt ; ls -1d ._view/* ) && sed -i \"s;/view/;/$original_view/;\" /opt/spack-env/z10_spack_environment.sh && echo \"# Needed for Perl applications\" >>/opt/spack-env/z10_spack_environment.sh && echo \"export PERL5LIB=$(eval ls -d /opt/._view/*/lib/5.*):$PERL5LIB\" >>/opt/spack-env/z10_spack_environment.sh && rm -rf /opt/view \nINFO[0025] Initializing snapshotter ... \nINFO[0025] Taking snapshot of full filesystem... \nINFO[0040] Cmd: docker-shell \nINFO[0040] Args: [cd /opt/spack-env && spack env activate --sh -d . >> /opt/spack-env/z10_spack_environment.sh && original_view=$( cd /opt ; ls -1d ._view/* ) && sed -i \"s;/view/;/$original_view/;\" /opt/spack-env/z10_spack_environment.sh && echo \"# Needed for Perl applications\" >>/opt/spack-env/z10_spack_environment.sh && echo \"export PERL5LIB=$(eval ls -d /opt/._view/*/lib/5.*):$PERL5LIB\" >>/opt/spack-env/z10_spack_environment.sh && rm -rf /opt/view] \nINFO[0040] Running: [docker-shell cd /opt/spack-env && spack env activate --sh -d . >> /opt/spack-env/z10_spack_environment.sh && original_view=$( cd /opt ; ls -1d ._view/* ) && sed -i \"s;/view/;/$original_view/;\" /opt/spack-env/z10_spack_environment.sh && echo \"# Needed for Perl applications\" >>/opt/spack-env/z10_spack_environment.sh && echo \"export PERL5LIB=$(eval ls -d /opt/._view/*/lib/5.*):$PERL5LIB\" >>/opt/spack-env/z10_spack_environment.sh && rm -rf /opt/view] \nls: cannot access '._view/*': No such file or directory\nerror building image: error building stage: failed to execute command: waiting for process to exit: exit status 2\n"}]}
at io.seqera.wave.cli.Client.awaitImage(Client.java:174)
at io.seqera.wave.cli.App.run(App.java:411)
at io.seqera.wave.cli.App.main(App.java:217)
https://github.com/Emiller88/auto-wave/pull/5
That would be helpful
git clone https://github.com/Emiller88/auto-wave.git
cd auto-wave
git checkout -b spack
wave --spack-file star/spack.yml --await
There is an error because of the spack version (Note: same is happening for singularity builds)
==> Fetching https://binaries.spack.io/releases/v0.20/build_cache/_pgp/spack-public-binary-key.pub
==> Error: you need a newer Spack version to read the buildcache index for the following mirror: 'https://binaries.spack.io/releases/v0.20'. The expected DB version is '6', but '7' was found.
@marcodelapierre @bebosudo is there an update or change in buildcache?
This should be solved once we implement the architecture detailed in https://github.com/seqeralabs/devops-backlog/issues/269.
This is still happening FYI https://github.com/nf-core/modules/actions/runs/7944492424/job/21690068753
Thanks for your message Edmund.
This is expected at this stage. Spack in Wave development is in progress, and as such we cannot guarantee it is operational.
We are working on releasing feature milestones for Spack in Wave later this year.
