bioconda-recipes
bioconda-recipes copied to clipboard
bioconda
Build Bioconductor-BiocGenerics for linux-aarch64
Describe your pull request here
Please read the guidelines for Bioconda recipes before opening a pull request (PR).
General instructions
- If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
- New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
- PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the
@BiocondaBot please add labelcommand. - Please post questions on Gitter or ping
@bioconda/corein a comment.
Instructions for avoiding API, ABI, and CLI breakage issues
Conda is able to record and lock (a.k.a. pin) dependency versions used at build time of other recipes.
This way, one can avoid that expectations of a downstream recipe with regards to API, ABI, or CLI are violated by later changes in the recipe.
If not already present in the meta.yaml, make sure to specify run_exports (see here for the rationale and comprehensive explanation).
Add a run_exports section like this:
build:
run_exports:
- ...
with ... being one of:
| Case | run_exports statement |
|---|---|
| semantic versioning | {{ pin_subpackage("myrecipe", max_pin="x") }} |
| semantic versioning (0.x.x) | {{ pin_subpackage("myrecipe", max_pin="x.x") }} |
| known breakage in minor versions | {{ pin_subpackage("myrecipe", max_pin="x.x") }} (in such a case, please add a note that shortly mentions your evidence for that) |
| known breakage in patch versions | {{ pin_subpackage("myrecipe", max_pin="x.x.x") }} (in such a case, please add a note that shortly mentions your evidence for that) |
| calendar versioning | {{ pin_subpackage("myrecipe", max_pin=None) }} |
while replacing "myrecipe" with either name if a name|lower variable is defined in your recipe or with the lowercase name of the package in quotes.
Bot commands for PR management
Please use the following BiocondaBot commands:
Everyone has access to the following BiocondaBot commands, which can be given in a comment:
@BiocondaBot please update |
Merge the master branch into a PR. |
@BiocondaBot please add label |
Add the please review & merge label. |
@BiocondaBot please fetch artifacts |
Post links to CI-built packages/containers. You can use this to test packages locally. |
Note that the @BiocondaBot please merge command is now depreciated. Please just squash and merge instead.
Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.
@daler @aliciaaevans I need your help here! The build on Linux ARM64 here failed with:
...
08:41:15 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:15] DEBU Pull Image [involucro/tool:latest][0m
08:41:16 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:16] DEBU Created container [a0ce85569d72 step-68d670006b], starting it[0m
08:41:16 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:16] DEBU Container [a0ce85569d72 step-68d670006b] started, waiting for completion[0m
08:41:16 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:16] SERR exec /involucro: exec format error[0m
08:41:16 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:16] ERRO Task processing failed: Unexpected exit code [1] of container [a0ce85569d72 step-68d670006b], container preserved[0m
.08:41:36 [32mBIOCONDA INFO[0m (OUT) .[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) Traceback (most recent call last):[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/bin/mulled-build", line 10, in <module>[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) sys.exit(main())[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/galaxy/tools/deps/mulled/mulled_build.py", line 414, in main[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) sys.exit(mull_targets(targets, **args_to_mull_targets_kwds(args)))[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/galaxy/tools/deps/mulled/mulled_build.py", line 243, in mull_targets[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) ret = involucro_context.exec_command(involucro_args)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/galaxy/tools/deps/mulled/mulled_build.py", line 287, in exec_command[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) shutil.rmtree('./build')[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 718, in rmtree[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) _rmtree_safe_fd(fd, path, onerror)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) _rmtree_safe_fd(dirfd, fullname, onerror)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) _rmtree_safe_fd(dirfd, fullname, onerror)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) _rmtree_safe_fd(dirfd, fullname, onerror)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) [Previous line repeated 2 more times][0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) onerror(os.unlink, fullname, sys.exc_info())[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) File "/opt/mambaforge/envs/bioconda/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) os.unlink(entry.name, dir_fd=topfd)[0m
08:41:36 [32mBIOCONDA INFO[0m (ERR) PermissionError: [Errno 13] Permission denied: 'collect2'[0m
08:41:36 [31mBIOCONDA ERROR[0m COMMAND FAILED (exited with 1): mulled-build build-and-test bioconductor-biocgenerics=0.48.1--r43hdfd78af_3 -n biocontainers --test bash -c '/usr/local/env-execute true && . /usr/local/env-activate.sh && Rscript -e "library('"'"'BiocGenerics'"'"')"' --channels conda-forge,file:///opt/mambaforge/envs/bioconda/conda-bld,bioconda,defaults --involucro-path /opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/involucro[0m
Traceback (most recent call last):
File "/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/pkg_test.py", line 180, in test_package
p = utils.run(cmd, env=env, cwd=d, mask=False, live=live_logs)
File "/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/utils.py", line 697, in run
raise sp.CalledProcessError(returncode, masked_cmds, output=output)
subprocess.CalledProcessError: Command '['mulled-build', 'build-and-test', 'bioconductor-biocgenerics=0.48.1--r43hdfd78af_3', '-n', 'biocontainers', '--test', 'bash -c \'/usr/local/env-execute true && . /usr/local/env-activate.sh && Rscript -e "library(\'"\'"\'BiocGenerics\'"\'"\')"\'', '--channels', 'conda-forge,file:///opt/mambaforge/envs/bioconda/conda-bld,bioconda,defaults', '--involucro-path', '/opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/involucro']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
...
https://hub.docker.com/r/involucro/tool/tags has only linux/amd64 image!
In addition https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tool_util/deps/mulled/mulled_build.py#L93-L98 uses only Linux AMD64 and Mac Intel binaries.
According to https://github.com/bioconda/bioconda-utils/blob/18f988d70966f6f6296170d96cc1ced51ad10392/bioconda_utils/upload.py#L80-L81 Galaxy will use a local involucro binary if the path is provided! So, this seems to be fine!
It seems involucro/tool:latest is pulled by https://github.com/involucro/involucro/blob/2255925a5a3d3ba4da9b45892399b4576ee4ec3d/internal/wrap.go#L314
It appears that involucro/tool:latest is used only as a fallback when the local execution has failed!
The local execution failed with
08:41:14 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:14] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0][0m
08:41:15 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:15] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied[0m
08:41:15 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:41:15] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution[0m
The actual problem is that there is no licenses/ folder in https://assets.ubuntu.com/v1/0cef8205-ubuntu-font-family-0.83.zip
https://github.com/conda-forge/font-ttf-ubuntu-feedstock/blob/main/recipe/meta.yaml
It fails while involucro is packing: https://github.com/involucro/involucro/blob/2255925a5a3d3ba4da9b45892399b4576ee4ec3d/internal/wrap.go#L136 but I cannot figure out yet where this licences/ comes from!
https://github.com/conda-forge/font-ttf-ubuntu-feedstock/blob/main/recipe/build.sh#L6 installs only the *.ttf files
It behaves the same way on linux-x64 at Azure: https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=52440&view=logs&jobId=e14e69ff-a0ae-55c4-b71d-229b239cfb2f&j=e14e69ff-a0ae-55c4-b71d-229b239cfb2f&t=4dddc55b-10bc-50fc-ac68-7ff899b7a030
https://dev.azure.com/bioconda/25074ded-3133-43ca-af80-895ab87b53cb/_apis/build/builds/52440/logs/26
But there the remote execution via involucro/tool helps:
...
2024-03-05T08:53:03.1049440Z 08:52:17 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:17] DEBU Image [quay.io/bioconda/base-glibc-busybox-bash:3.0] not present, pulling it[0m
2024-03-05T08:53:03.1050058Z 08:52:17 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:17] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0][0m
2024-03-05T08:53:03.1050615Z 08:52:19 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:19] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied[0m
2024-03-05T08:53:03.1051306Z 08:52:19 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:19] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution[0m
2024-03-05T08:53:03.1051872Z 08:52:19 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:19] DEBU Creating container [step-78c5b66fff][0m
2024-03-05T08:53:03.1052286Z 08:52:19 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:19] DEBU Image [involucro/tool:latest] not present, pulling it[0m
2024-03-05T08:53:03.1052725Z 08:52:19 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:19] DEBU Pull Image [involucro/tool:latest][0m
2024-03-05T08:53:03.1053153Z 08:52:20 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:20] DEBU Created container [0933cf8eee5b step-78c5b66fff], starting it[0m
2024-03-05T08:53:03.1053630Z 08:52:20 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:20] DEBU Container [0933cf8eee5b step-78c5b66fff] started, waiting for completion[0m
2024-03-05T08:53:03.1054086Z 08:52:20 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:20] SERR [Mar 5 08:52:20] DEBU Creating container [step-655c09cd6a][0m
2024-03-05T08:53:03.1054540Z 08:52:23 [32mBIOCONDA INFO[0m (ERR) [Mar 5 08:52:23] SERR [Mar 5 08:52:23] DEBU Packing succeeded[0m
...
What is h77eed37_1 in the table below ?
2024-03-05T08:42:11.3234834Z Package Version Build Channel Size
2024-03-05T08:42:11.3248305Z ──────────────────────────────────────────────────────────────────────────────────────────────────
2024-03-05T08:42:11.3248641Z Install:
2024-03-05T08:42:11.3249131Z ──────────────────────────────────────────────────────────────────────────────────────────────────
2024-03-05T08:42:11.3249333Z
2024-03-05T08:42:11.3254437Z + font-ttf-ubuntu 0.83 h77eed37_1 conda-forge/noarch 2MB
h77eed37 looks like a SHA. And 1 looks like a build number.
But I don't see a commit at https://github.com/conda-forge/font-ttf-ubuntu-feedstock/commits/main/ with similar SHA
Installing conda install font-ttf-ubuntu on my local Linux ARM64 machine prints:
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
It is the same version (0.83) but a different build.
I am trying to find out where this licenses/ comes from in Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied
I tried to reproduce the mulled-build locally:
- x86_64
$ conda create -n biotest
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 24.1.2
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/martin/devel/miniconda3/envs/biotest
Proceed ([y]/n)?
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate biotest
#
# To deactivate an active environment, use
#
# $ conda deactivate
$ conda activate biotest
$ conda install font-ttf-ubuntu (biotest)
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 24.1.2
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/martin/devel/miniconda3/envs/biotest
added / updated specs:
- font-ttf-ubuntu
The following packages will be downloaded:
package | build
---------------------------|-----------------
font-ttf-ubuntu-0.83 | h77eed37_1 1.5 MB conda-forge
------------------------------------------------------------
Total: 1.5 MB
The following NEW packages will be INSTALLED:
font-ttf-ubuntu conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
Proceed ([y]/n)?
Downloading and Extracting Packages
font-ttf-ubuntu-0.83 | 1.5 MB | ########################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
$ ll /home/martin/devel/miniconda3/envs/biotest/fonts/ (biotest)
.rw-rw-r-- martin martin 326 KB Sat Nov 25 22:30:15 2023 Ubuntu-B.ttf
.rw-rw-r-- martin martin 349 KB Sat Nov 25 22:30:15 2023 Ubuntu-BI.ttf
.rw-rw-r-- martin martin 342 KB Sat Nov 25 22:30:15 2023 Ubuntu-C.ttf
.rw-rw-r-- martin martin 406 KB Sat Nov 25 22:30:15 2023 Ubuntu-L.ttf
.rw-rw-r-- martin martin 400 KB Sat Nov 25 22:30:15 2023 Ubuntu-LI.ttf
.rw-rw-r-- martin martin 333 KB Sat Nov 25 22:30:15 2023 Ubuntu-M.ttf
.rw-rw-r-- martin martin 358 KB Sat Nov 25 22:30:15 2023 Ubuntu-MI.ttf
.rw-rw-r-- martin martin 346 KB Sat Nov 25 22:30:15 2023 Ubuntu-R.ttf
.rw-rw-r-- martin martin 377 KB Sat Nov 25 22:30:15 2023 Ubuntu-RI.ttf
.rw-rw-r-- martin martin 234 KB Sat Nov 25 22:30:15 2023 Ubuntu-Th.ttf
.rw-rw-r-- martin martin 187 KB Sat Nov 25 22:30:15 2023 UbuntuMono-B.ttf
.rw-rw-r-- martin martin 211 KB Sat Nov 25 22:30:15 2023 UbuntuMono-BI.ttf
.rw-rw-r-- martin martin 201 KB Sat Nov 25 22:30:15 2023 UbuntuMono-R.ttf
.rw-rw-r-- martin martin 205 KB Sat Nov 25 22:30:15 2023 UbuntuMono-RI.ttf
/tmp via ☕ v17.0.10 via 🅒 biotest
❯ ll /home/martin/devel/miniconda3/envs/biotest/conda-meta/ (biotest)
.rw-rw-r-- martin martin 5.7 KB Wed Mar 6 12:01:38 2024 font-ttf-ubuntu-0.83-h77eed37_1.json
.rw-rw-r-- martin martin 329 B Wed Mar 6 12:01:38 2024 history
$ pip install galaxy-tool-util
...
$ conda install involucro (biotest)
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/martin/devel/miniconda3/envs/biotest
added / updated specs:
- involucro
The following packages will be downloaded:
package | build
---------------------------|-----------------
involucro-1.1.2 | he881be0_4 2.4 MB bioconda
------------------------------------------------------------
Total: 2.4 MB
The following NEW packages will be INSTALLED:
involucro bioconda/linux-64::involucro-1.1.2-he881be0_4
Proceed ([y]/n)?
Downloading and Extracting Packages
involucro-1.1.2 | 2.4 MB | ########################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
$ which involucro (biotest)
/home/martin/devel/miniconda3/envs/biotest/bin/involucro
$ mulled-build build-and-test 'font-ttf-ubuntu=0.83' --involucro-path /home/martin/devel/miniconda3/envs/biotest/bin/involucro --verbose (biotest)
Executing: /home/martin/devel/miniconda3/envs/biotest/bin/involucro -v=3 -f /home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,bioconda -set TARGETS=font-ttf-ubuntu=0.83 -set REPO=quay.io/biocontainers/font-ttf-ubuntu:0.83 -set BINDS=build/dist:/usr/local/ -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:latest -set CONDA_BIN=conda build-and-test
.[Mar 6 12:04:44] DEBU Run file [/home/martin/.local/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Mar 6 12:04:44] INFO Invoke Task [build]
[Mar 6 12:04:44] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[rm -rf /data/dist]]
[Mar 6 12:04:44] DEBU Creating container [step-ddb3834f95]
[Mar 6 12:04:44] DEBU Image [quay.io/condaforge/mambaforge:latest] not present, pulling it
[Mar 6 12:04:44] DEBU Pull Image [quay.io/condaforge/mambaforge:latest]
[Mar 6 12:04:47] PRGS {Pulling from condaforge/mambaforge {0 0} }
...
[Mar 6 12:05:04] PRGS {Digest: sha256:478955c6fa52b5afd6115c0e8f744642494c5e1d219d26bf2fe438c0e870f9ba {0 0} }
[Mar 6 12:05:04] PRGS {Status: Downloaded newer image for quay.io/condaforge/mambaforge:latest {0 0} }
[Mar 6 12:05:05] DEBU Created container [d6c1348c797e step-ddb3834f95], starting it
[Mar 6 12:05:05] DEBU Container [d6c1348c797e step-ddb3834f95] started, waiting for completion
[Mar 6 12:05:05] DEBU Container [d6c1348c797e step-ddb3834f95] completed with exit code [0] as expected
[Mar 6 12:05:05] DEBU Container [d6c1348c797e step-ddb3834f95] removed
[Mar 6 12:05:05] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[/bin/sh -c conda install -c 'conda-forge' -c 'bioconda' 'font-ttf-ubuntu=0.83' --strict-channel-priority -p /usr/local --copy --yes --quiet]]
[Mar 6 12:05:05] DEBU Creating container [step-54f1a7db56]
[Mar 6 12:05:05] DEBU Created container [bee633a81f00 step-54f1a7db56], starting it
[Mar 6 12:05:05] DEBU Container [bee633a81f00 step-54f1a7db56] started, waiting for completion
[Mar 6 12:05:06] SOUT Channels:
[Mar 6 12:05:06] SOUT - conda-forge
[Mar 6 12:05:06] SOUT - bioconda
[Mar 6 12:05:06] SOUT Platform: linux-64
.[Mar 6 12:05:48] SOUT Collecting package metadata (repodata.json): ...working... done
[Mar 6 12:05:48] SOUT Solving environment: ...working... done
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT ## Package Plan ##
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT environment location: /usr/local
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT added / updated specs:
[Mar 6 12:05:49] SOUT - font-ttf-ubuntu=0.83
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT The following packages will be downloaded:
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT package | build
[Mar 6 12:05:49] SOUT ---------------------------|-----------------
[Mar 6 12:05:49] SOUT font-ttf-ubuntu-0.83 | h77eed37_1 1.5 MB conda-forge
[Mar 6 12:05:49] SOUT ------------------------------------------------------------
[Mar 6 12:05:49] SOUT Total: 1.5 MB
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT The following NEW packages will be INSTALLED:
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT font-ttf-ubuntu conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT
[Mar 6 12:05:49] SOUT Preparing transaction: ...working... done
[Mar 6 12:05:49] SOUT Verifying transaction: ...working... done
[Mar 6 12:05:49] SOUT Executing transaction: ...working... done
[Mar 6 12:05:49] DEBU Container [bee633a81f00 step-54f1a7db56] completed with exit code [0] as expected
[Mar 6 12:05:49] DEBU Container [bee633a81f00 step-54f1a7db56] removed
[Mar 6 12:05:49] STEP Wrap [build/dist] as [quay.io/biocontainers/font-ttf-ubuntu:0.83]
[Mar 6 12:05:49] DEBU Creating container [step-2f7e39d51a]
[Mar 6 12:05:49] DEBU Image [quay.io/bioconda/base-glibc-busybox-bash:latest] not present, pulling it
[Mar 6 12:05:49] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:latest]
[Mar 6 12:05:52] PRGS {Pulling from bioconda/base-glibc-busybox-bash {0 0} }
...
[Mar 6 12:05:56] PRGS {Digest: sha256:cec304688c4ec33b310b2181f0bb2d5f6b805c92784feec39543c922c38db853 {0 0} }
[Mar 6 12:05:56] PRGS {Status: Downloaded newer image for quay.io/bioconda/base-glibc-busybox-bash:latest {0 0} }
[Mar 6 12:05:56] DEBU Packing succeeded
[Mar 6 12:05:56] INFO Invoke Task [singularity]
[Mar 6 12:05:56] WARN Invoked Stepless Task [singularity]
[Mar 6 12:05:56] INFO Invoke Task [cleanup]
[Mar 6 12:05:56] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[rm -rf /data/dist]]
[Mar 6 12:05:56] DEBU Creating container [step-496a937dec]
[Mar 6 12:05:56] DEBU Created container [d6477bfecd66 step-496a937dec], starting it
[Mar 6 12:05:56] DEBU Container [d6477bfecd66 step-496a937dec] started, waiting for completion
[Mar 6 12:05:56] DEBU Container [d6477bfecd66 step-496a937dec] completed with exit code [0] as expected
[Mar 6 12:05:56] DEBU Container [d6477bfecd66 step-496a937dec] removed
[Mar 6 12:05:56] INFO Invoke Task [test]
[Mar 6 12:05:56] STEP Run image [quay.io/biocontainers/font-ttf-ubuntu:0.83] with command [[]]
[Mar 6 12:05:56] DEBU Creating container [step-08294e7104]
[Mar 6 12:05:56] DEBU Created container [a79a68e2cbdd step-08294e7104], starting it
[Mar 6 12:05:56] DEBU Container [a79a68e2cbdd step-08294e7104] started, waiting for completion
[Mar 6 12:05:57] DEBU Container [a79a68e2cbdd step-08294e7104] completed with exit code [0] as expected
[Mar 6 12:05:57] DEBU Container [a79a68e2cbdd step-08294e7104] removed
No issues with licenses/ !
mulled-build fails on my Linux ARM64 machine with:
Executing: /home/mgrigorov/miniconda3/envs/font-ubuntu/bin/involucro -v=3 -f /home/mgrigorov/miniconda3/envs/font-ubuntu/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,bioconda -set TARGETS=font-ttf-ubuntu=0.83 -set REPO=quay.io/biocontainers/font-ttf-ubuntu:0.83 -set BINDS=build/dist:/usr/local/ -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:latest -set CONDA_BIN=conda build-and-test
.[Mar 6 13:05:25] DEBU Run file [/home/mgrigorov/miniconda3/envs/font-ubuntu/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Mar 6 13:05:25] INFO Invoke Task [build]
[Mar 6 13:05:25] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[rm -rf /data/dist]]
[Mar 6 13:05:25] DEBU Creating container [step-b5fea6f6cf]
[Mar 6 13:05:25] DEBU Created container [fb0a18fde0d9 step-b5fea6f6cf], starting it
[Mar 6 13:05:26] DEBU Container [fb0a18fde0d9 step-b5fea6f6cf] started, waiting for completion
[Mar 6 13:05:26] DEBU Container [fb0a18fde0d9 step-b5fea6f6cf] completed with exit code [0] as expected
[Mar 6 13:05:26] DEBU Container [fb0a18fde0d9 step-b5fea6f6cf] removed
[Mar 6 13:05:26] STEP Run image [quay.io/condaforge/mambaforge:latest] with command [[/bin/sh -c conda install -c 'conda-forge' -c 'bioconda' 'font-ttf-ubuntu=0.83' --strict-channel-priority -p /usr/local --copy --yes --quiet]]
[Mar 6 13:05:26] DEBU Creating container [step-24458744b0]
[Mar 6 13:05:26] DEBU Created container [af7fc35484d1 step-24458744b0], starting it
[Mar 6 13:05:26] DEBU Container [af7fc35484d1 step-24458744b0] started, waiting for completion
[Mar 6 13:05:27] SOUT Channels:
[Mar 6 13:05:27] SOUT - conda-forge
[Mar 6 13:05:27] SOUT - bioconda
[Mar 6 13:05:27] SOUT Platform: linux-aarch64
[Mar 6 13:06:13] SOUT Collecting package metadata (repodata.json): ...working... done
[Mar 6 13:06:13] SOUT Solving environment: ...working... done
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT ## Package Plan ##
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT environment location: /usr/local
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT added / updated specs:
[Mar 6 13:06:13] SOUT - font-ttf-ubuntu=0.83
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT The following packages will be downloaded:
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT package | build
[Mar 6 13:06:13] SOUT ---------------------------|-----------------
[Mar 6 13:06:13] SOUT font-ttf-ubuntu-0.83 | h77eed37_1 1.5 MB conda-forge
[Mar 6 13:06:13] SOUT ------------------------------------------------------------
[Mar 6 13:06:13] SOUT Total: 1.5 MB
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT The following NEW packages will be INSTALLED:
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT font-ttf-ubuntu conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT
[Mar 6 13:06:13] SOUT Preparing transaction: ...working... done
[Mar 6 13:06:13] SOUT Verifying transaction: ...working... done
[Mar 6 13:06:13] SOUT Executing transaction: ...working... done
[Mar 6 13:06:14] DEBU Container [af7fc35484d1 step-24458744b0] completed with exit code [0] as expected
[Mar 6 13:06:14] DEBU Container [af7fc35484d1 step-24458744b0] removed
[Mar 6 13:06:14] STEP Wrap [build/dist] as [quay.io/biocontainers/font-ttf-ubuntu:0.83]
[Mar 6 13:06:14] DEBU Creating container [step-70211fda12]
[Mar 6 13:06:14] DEBU Packing failed due to open build/dist/conda-meta: permission denied
[Mar 6 13:06:14] WARN Local execution errorred with error [open build/dist/conda-meta: permission denied], retrying with remote execution
[Mar 6 13:06:14] DEBU Creating container [step-aa4b30dfd2]
[Mar 6 13:06:14] DEBU Created container [76cd70e11125 step-aa4b30dfd2], starting it
[Mar 6 13:06:14] DEBU Container [76cd70e11125 step-aa4b30dfd2] started, waiting for completion
[Mar 6 13:06:14] SERR standard_init_linux.go:224: exec user process caused "exec format error"
[Mar 6 13:06:14] ERRO Task processing failed: Unexpected exit code [1] of container [76cd70e11125 step-aa4b30dfd2], container preserved
Similar error (Permission denied), but not exactly the same.
I was able to reproduce it on Linux x86_64 too! But it is random - sometimes it passes, sometimes it fails with permission denied. On Linux aarch64 it fails consistently!
conda create -n test123
conda activate test123
conda install font-ttf-ubuntu involucro
pip3 install galaxy-tool-util
mkdir /tmp/mulled && cd /tmp/mulled
mulled-build build 'font-ttf-ubuntu=0.83' --involucro-path $(which involucro) --verbose
ls -laR
ls -laR prints that the build/ is owned by my user (martin) but anything inside is owned by root (the user from DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:latest) and later the Involucro wrapping fails to read files from it:
ls -laR (biotest)
.:
total 76
drwxrwxr-x 3 martin martin 4096 Mar 6 15:42 ./
drwxrwxrwt 44 root root 65536 Mar 6 15:49 ../
drwxrwxr-x 3 martin martin 4096 Mar 6 15:47 build/
./build:
total 12
drwxrwxr-x 3 martin martin 4096 Mar 6 15:47 ./
drwxrwxr-x 3 martin martin 4096 Mar 6 15:42 ../
drwxr-xr-x 4 root root 4096 Mar 6 15:47 dist/
./build/dist:
total 16
drwxr-xr-x 4 root root 4096 Mar 6 15:47 ./
drwxrwxr-x 3 martin martin 4096 Mar 6 15:47 ../
drwxr-xr-x 2 root root 4096 Mar 6 15:47 conda-meta/
drwxr-xr-x 2 root root 4096 Mar 6 15:47 fonts/
./build/dist/conda-meta:
total 20
drwxr-xr-x 2 root root 4096 Mar 6 15:47 ./
drwxr-xr-x 4 root root 4096 Mar 6 15:47 ../
-rw-r--r-- 1 root root 5740 Mar 6 15:47 font-ttf-ubuntu-0.83-h77eed37_1.json
-rw-r--r-- 1 root root 292 Mar 6 15:47 history
./build/dist/fonts:
total 4316
drwxr-xr-x 2 root root 4096 Mar 6 15:47 ./
drwxr-xr-x 4 root root 4096 Mar 6 15:47 ../
-rw-r--r-- 1 root root 356980 Nov 25 22:30 Ubuntu-BI.ttf
-rw-r--r-- 1 root root 333612 Nov 25 22:30 Ubuntu-B.ttf
-rw-r--r-- 1 root root 350444 Nov 25 22:30 Ubuntu-C.ttf
-rw-r--r-- 1 root root 409608 Nov 25 22:30 Ubuntu-LI.ttf
-rw-r--r-- 1 root root 415552 Nov 25 22:30 Ubuntu-L.ttf
-rw-r--r-- 1 root root 366992 Nov 25 22:30 Ubuntu-MI.ttf
-rw-r--r-- 1 root root 216208 Nov 25 22:30 UbuntuMono-BI.ttf
-rw-r--r-- 1 root root 191400 Nov 25 22:30 UbuntuMono-B.ttf
-rw-r--r-- 1 root root 210216 Nov 25 22:30 UbuntuMono-RI.ttf
-rw-r--r-- 1 root root 205748 Nov 25 22:30 UbuntuMono-R.ttf
-rw-r--r-- 1 root root 341324 Nov 25 22:30 Ubuntu-M.ttf
-rw-r--r-- 1 root root 386440 Nov 25 22:30 Ubuntu-RI.ttf
-rw-r--r-- 1 root root 353824 Nov 25 22:30 Ubuntu-R.ttf
-rw-r--r-- 1 root root 240020 Nov 25 22:30 Ubuntu-Th.ttf
@bgruening From other issues I see you have experience with mulled-build. Do you have an idea how this is supposed to work ? Thank you!
I've found the issue!
On my x86_64 machine umask is 0002 and the others have an read+execute bit for the folders:
drwxr-xr-x 2 root root 4096 Mar 6 15:47 conda-meta/
drwxr-xr-x 2 root root 4096 Mar 6 15:47 fonts/
On my aarch64 machine the value is 0022 and the permissions look like:
drwxr-x--- 2 root root 80 Mar 6 13:39 conda-meta
drwxr-x--- 2 root root 320 Mar 6 13:39 fonts
ls: cannot open directory './build/dist/conda-meta': Permission denied
ls: cannot open directory './build/dist/fonts': Permission denied
The problem is a bit deeper:
x86_64:
docker run -it --rm quay.io/bioconda/base-glibc-busybox-bash:latest bash (biotest)
root@db1de61174f3:/# umask
0022
aarch64:
docker run -it --rm quay.io/bioconda/base-glibc-busybox-bash:latest bash (font-ubuntu)
root@878179165bf7:/# umask
0027
Same issue for quay.io/condaforge/mambaforge:latest
@daler ^^
I tried with custom mambaforge and base-glibc-busybox-bash images that set umask to 022 but it still fails the same way ...
Dockerfile.mambaforge:
FROM quay.io/condaforge/mambaforge:latest
ENV BASH_ENV /etc/profile
RUN echo "umask 022" >> /etc/profile # non-interactive
RUN echo "umask 022" >> /etc/bash.bashrc # interactive
$ docker build . -f Dockerfile.mambaforge -t my-mambaforge-aarch64:latest
$ docker run -it --rm my-mambaforge-aarch64:latest bash -c umask # non-interactive
$ docker run -it --rm my-mambaforge-aarch64:latest bash (+ `umask`) # interactive
Both return 0022!
Same for busybox.
$ export CONDA_IMAGE=my-mambaforge-aarch64:latest
$ export DEFAULT_BASE_IMAGE=my-base-glibc-busybox-bash-aarch64:latest
$ mulled-build build 'font-ttf-ubuntu=0.83' --involucro-path (which involucro) --verbose (font-ubuntu)
Executing: /home/mgrigorov/miniconda3/envs/font-ubuntu/bin/involucro -v=3 -f /home/mgrigorov/miniconda3/envs/font-ubuntu/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,bioconda -set TARGETS=font-ttf-ubuntu=0.83 -set REPO=quay.io/biocontainers/font-ttf-ubuntu:0.83 -set BINDS=build/dist:/usr/local/ -set DEST_BASE_IMAGE=my-base-glibc-busybox-bash-aarch64:latest -set CONDA_IMAGE=my-mambaforge-aarch64:latest -set CONDA_BIN=conda build
.[Mar 7 12:23:47] DEBU Run file [/home/mgrigorov/miniconda3/envs/font-ubuntu/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Mar 7 12:23:47] STEP Run image [my-mambaforge-aarch64:latest] with command [[rm -rf /data/dist]]
[Mar 7 12:23:47] DEBU Creating container [step-0eb06fff49]
[Mar 7 12:23:47] DEBU Created container [192dfb6f0c5b step-0eb06fff49], starting it
[Mar 7 12:23:47] DEBU Container [192dfb6f0c5b step-0eb06fff49] started, waiting for completion
[Mar 7 12:23:48] DEBU Container [192dfb6f0c5b step-0eb06fff49] completed with exit code [0] as expected
[Mar 7 12:23:48] DEBU Container [192dfb6f0c5b step-0eb06fff49] removed
[Mar 7 12:23:48] STEP Run image [my-mambaforge-aarch64:latest] with command [[/bin/sh -c conda install -c 'conda-forge' -c 'bioconda' 'font-ttf-ubuntu=0.83' --strict-channel-priority -p /usr/local --copy --yes --quiet]]
[Mar 7 12:23:48] DEBU Creating container [step-b1af40742e]
[Mar 7 12:23:48] DEBU Created container [de5870c61766 step-b1af40742e], starting it
[Mar 7 12:23:48] DEBU Container [de5870c61766 step-b1af40742e] started, waiting for completion
[Mar 7 12:23:49] SOUT Channels:
[Mar 7 12:23:49] SOUT - conda-forge
[Mar 7 12:23:49] SOUT - bioconda
[Mar 7 12:23:49] SOUT Platform: linux-aarch64
[Mar 7 12:24:35] SOUT Collecting package metadata (repodata.json): ...working... done
[Mar 7 12:24:35] SOUT Solving environment: ...working... done
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT ## Package Plan ##
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT environment location: /usr/local
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT added / updated specs:
[Mar 7 12:24:36] SOUT - font-ttf-ubuntu=0.83
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT The following packages will be downloaded:
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT package | build
[Mar 7 12:24:36] SOUT ---------------------------|-----------------
[Mar 7 12:24:36] SOUT font-ttf-ubuntu-0.83 | h77eed37_1 1.5 MB conda-forge
[Mar 7 12:24:36] SOUT ------------------------------------------------------------
[Mar 7 12:24:36] SOUT Total: 1.5 MB
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT The following NEW packages will be INSTALLED:
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT font-ttf-ubuntu conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_1
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT
[Mar 7 12:24:36] SOUT Preparing transaction: ...working... done
[Mar 7 12:24:36] SOUT Verifying transaction: ...working... done
[Mar 7 12:24:36] SOUT Executing transaction: ...working... done
[Mar 7 12:24:37] DEBU Container [de5870c61766 step-b1af40742e] completed with exit code [0] as expected
[Mar 7 12:24:37] DEBU Container [de5870c61766 step-b1af40742e] removed
[Mar 7 12:24:37] STEP Wrap [build/dist] as [quay.io/biocontainers/font-ttf-ubuntu:0.83]
[Mar 7 12:24:37] DEBU Creating container [step-b6c341175b]
[Mar 7 12:24:37] DEBU Packing failed due to open build/dist/conda-meta: permission denied
[Mar 7 12:24:37] WARN Local execution errorred with error [open build/dist/conda-meta: permission denied], retrying with remote execution
[Mar 7 12:24:37] DEBU Creating container [step-9a98b90279]
[Mar 7 12:24:37] DEBU Created container [2cd55050a028 step-9a98b90279], starting it
[Mar 7 12:24:37] DEBU Container [2cd55050a028 step-9a98b90279] started, waiting for completion
[Mar 7 12:24:37] SERR standard_init_linux.go:224: exec user process caused "exec format error"
[Mar 7 12:24:37] ERRO Task processing failed: Unexpected exit code [1] of container [2cd55050a028 step-9a98b90279], container preserved
$ ls -laR
...
drwxr-x--- 2 root root 80 Mar 7 12:24 conda-meta
drwxr-x--- 2 root root 320 Mar 7 12:24 fonts
ls: cannot open directory './build/dist/conda-meta': Permission denied
ls: cannot open directory './build/dist/fonts': Permission denied
docker run -it --rm my-mambaforge-aarch64:latest sh (font-ubuntu)
# umask
0027
For some reason sh (dash) does not read /etc/profile ...
The problem is in mulled-build-tool's default invfile.lua - https://github.com/galaxyproject/galaxy/blob/bc0d075d552cc10d9c452e9e0a4595b3e079301e/lib/galaxy/tool_util/deps/mulled/invfile.lua#L90
It uses .run('/bin/sh', '-c', preinstall and this ignores umask completely !
It seems the current blocker is https://github.com/bioconda/bioconda-utils/blob/18f988d70966f6f6296170d96cc1ced51ad10392/bioconda_utils/build.py#L125-L129
There is no way to use a custom base_image for the mulled-build ...
@martin-g thank you for the epic troubleshooting! To make sure I understand this all: the busybox aarch64 container and the x86_64 container disagree in their umasks?
And secondarily, you can't patch in a custom container for mulled-build to use because the container is hard-coded in bioconda-utils.
IIRC the original rational for the extended base image was to deal with font stuff. Does it work if you configure the debian extended base image in the recipe?
the busybox aarch64 container and the x86_64 container disagree in their umasks?
Correct!
The difference is only when /bin/sh is used. In this case neither /etc/profile nor pam_umask is taken into account and the umask on aarch64 is 027. On x86_64 it is 022 - this makes it possible one OS user to list the files created by another user.
And secondarily, you can't patch in a custom container for mulled-build to use because the container is hard-coded in bioconda-utils.
I was just trying to "patch" the environment so I can see a green build. Once the required changes are clear I was going to suggest to change bioconda-containers (both create-env and base-**-bash images.
Until https://github.com/galaxyproject/galaxy/issues/17631 is resolved my workaround is to replace the stock /bin/sh with a simple Shell script that delegates to /bin/bash. A symbolic link does not work because Bash checks args[0] and behaves differently (backward compat mode) when called as sh.
IIRC the original rational for the extended base image was to deal with font stuff. Does it work if you configure the debian extended base image in the recipe?
I will test it!
Update: it failed - https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/110365/workflows/3e705dd3-a409-455a-a6f3-cbb67cf53f32/jobs/181190?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary
08:36:25 [32mBIOCONDA INFO[0m (COMMAND) mulled-build build-and-test bioconductor-biocgenerics=0.48.1--r43hdfd78af_3 -n biocontainers --test bash -c '/usr/local/env-execute true && . /usr/local/env-activate.sh && Rscript -e "library('"'"'BiocGenerics'"'"')"' --channels conda-forge,file:///opt/mambaforge/envs/bioconda/conda-bld,bioconda,defaults --involucro-path /opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/involucro[0m
08:36:26 [32mBIOCONDA INFO[0m (OUT) /opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/involucro -v=2 -f /opt/mambaforge/envs/bioconda/lib/python3.8/site-packages/galaxy/tools/deps/mulled/invfile.lua -set CHANNELS='conda-forge,file:///opt/mambaforge/envs/bioconda/conda-bld,bioconda,defaults' -set TARGETS='bioconductor-biocgenerics=0.48.1=r43hdfd78af_3' -set REPO='quay.io/biocontainers/bioconductor-biocgenerics:0.48.1--r43hdfd78af_3' -set BINDS='build/dist:/usr/local/,/opt/mambaforge/envs/bioconda/conda-bld:/opt/mambaforge/envs/bioconda/conda-bld' -set DEST_BASE_IMAGE='quay.io/bioconda/base-glibc-debian-bash:3.0' -set CONDA_IMAGE='bioconda-mulled-aarch64:latest' -set TEST='bash -c '"'"'/usr/local/env-execute true && . /usr/local/env-activate.sh && Rscript -e "library('"'"'"'"'"'"'"'"'BiocGenerics'"'"'"'"'"'"'"'"')"'"'"'' build-and-test[0m
Also (aarch64):
mgrigorov in 🌐 euler-arm-22 in /tmp/mulled took 33s
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 sh -c umask
0027
mgrigorov in 🌐 euler-arm-22 in /tmp/mulled
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 bash -c umask
0027
mgrigorov in 🌐 euler-arm-22 in /tmp/mulled
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 sh
# umask
0027
#
mgrigorov in 🌐 euler-arm-22 in /tmp/mulled took 3s
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 bash
root@f819f23d2830:/# umask
0027
root@f819f23d2830:/#
exit
and on x86_64:
/tmp/mulled via 🅒 biotest took 9s
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 sh -c umask
0022
/tmp/mulled via 🅒 biotest
❯ docker run -it --rm quay.io/bioconda/base-glibc-debian-bash:3.0 bash -c umask
0022
Waiting for mulled-build-tool v24 to be released. There we could use a custom invfile.lua that will pass -l to /bin/sh that should resolve the issue with the umask.
@martin-g for the time being you could use something like python -m pip install git+https://[email protected]/galaxyproject/galaxy.git@dev#subdirectory=packages/tool_util
@bgruening The mulled build passed ! Thank you for this hint !
Currently bioconda-utils uses galaxy-tool-util 18.9.1 and newer: https://github.com/bioconda/bioconda-utils/blob/9f6df10bfecd048956acc80e7bb3d57952585529/bioconda_utils/bioconda_utils-requirements.txt#L55 I hope it could be updated directly to 23.2 once it is released !
OK, after reading all @martin-g's notes, doing some digging, and discussing with @daler, here's what I think is going on.
-
The LICENCE.txt file does have those owner-only permissions at the source. I'll see about a PR on the feedstock to make that file readable in the build. That won't fix any other files in other packages that might have a similar issue, but it would unblock many/all of these linked PRs. Not sure on the timeline for that.
-
The underlying issue appears that to be that the build is occurring inside containers but the wrap command is outside and so the user is not root. (I think that agrees with what @mbargull suggested.) See https://github.com/galaxyproject/galaxy/blob/21a57d24bc00a6c38103f2338ab0c49ebabbce20/lib/galaxy/tool_util/deps/mulled/invfile.lua#L97. Since on amd64 it does the "remote execution" fallback in an
involucro/tooldocker container, it is able to succeed. The CircleCI build throws theexec /involucro: exec format errorwhich is probably due to the architecture mismatch. There is no arm64 container at https://hub.docker.com/r/involucro/tool/tags.
@bgruening does this seem right to you? If there was an arm involucro/tool image, I think it would work around this issue for linux-arm64 the same way as linux-64. It's not ideal but that could buy some time for a longer term solution. Would getting that involve a lot of effort? (Since the latest one was from 5 years ago.) Do you have any suggestions that could fix the issue without having to do the fallback to remote execution?
- The LICENCE.txt file does have those owner-only permissions at the source. I'll see about a PR on the feedstock to make that file readable in the build. That won't fix any other files in other packages that might have a similar issue, but it would unblock many/all of these linked PRs. Not sure on the timeline for that.
One question here:
The error is failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt but the licence is in info folder:
$ tar xvf pkg-font-ttf-ubuntu-0.83-h77eed37_1.tar.zst
fonts/UbuntuMono-RI.ttf
fonts/Ubuntu-Th.ttf
fonts/Ubuntu-BI.ttf
...
info/licenses/LICENCE.txt
...
Are those the same files ? I guess they are but I am not sure due to the missing info/ folder in the error.
I hope we fixed it with https://github.com/conda-forge/font-ttf-ubuntu-feedstock/pull/3 !
@aliciaaevans this all makes perfect sense to me. Thanks all for digging this up. I'm wondering if we can use https://quay.io/biocontainers/involucro/ (and create ARM builds), I will also try to create ARM build for the fallback container. Need time to investigate a bit.
I hope we fixed it with conda-forge/font-ttf-ubuntu-feedstock#3 !
Cool! You beat me to it. 🥇
I think we probably should also still move forward with getting the fallback working for ARM in case of other packages having this issue.
@aliciaaevans this all makes perfect sense to me. Thanks all for digging this up. I'm wondering if we can use https://quay.io/biocontainers/involucro/ (and create ARM builds), I will also try to create ARM build for the fallback container. Need time to investigate a bit.
Great, thanks @bgruening! The fallback image looks to be hard-coded in involucro, but I guess that could be made configurable?