build:: conveyor failed to get: unable to retrieve auth token

[adriansev]

Version of Singularity 3.9.1-1.fc35

Describe the bug

FATAL:   While performing build: conveyor failed to get: unable to retrieve auth token: invalid username/password: unauthorized: HTTP Basic: Access denied

and the image is cached local, as is freshly build, and the remote does not need auth either way

podman image ls | grep alienpy
gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy        latest      218387a06b35  31 minutes ago  276 MB
gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy        1.3.6       6a740c10c0fe  32 minutes ago  276 MB

just trying to do https://gitlab.cern.ch/asevcenc/alienpy.cont/-/blob/master/build_singularity#L30

To Reproduce trying to build: https://gitlab.cern.ch/asevcenc/alienpy.cont/-/blob/master/alienpy.sing

Expected behavior to build

OS / Linux Distribution fedora 35

Installation Method building the rpm with rpmbuild

[adriansev]

so i solved this, after logout of the endpoint, deleting all credentials and login again (even if it should not have been necessary .. ) but the problem remains as the pull request auth and it should not. thank you!

[dtrudg]

Hi @adriansev - if you have credentials stored for an OCI registry, then a pull will always attempt to use them. This is mirroring the behavior of docker etc. It's important now because with the Docker Hub rate limiting we want pulls to be counted as authenticated if a user has an account with the higher rate limits.

I would therefore expect that the pull should work with a logout / credential deletion.... and without logging back in again. I'm not sure if this was the case?

[adriansev]

hi @dtrudg and sorry for the late answer! (and thanks for looking into this!) actually i had to login to be able to pull from my public project repository .. so, after making sure that i have no credentials saved in ~/.singularity i tried:

singularity pull docker://gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy:latest
FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy:latest: unable to retrieve auth token: invalid username/password: unauthorized: HTTP Basic: Access denied

then i did:

singularity remote login -u asevcenc -p "$(< ~/gitlab_cern.token)" docker://gitlab-registry.cern.ch

and finally it worked:

singularity pull docker://gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 04c2e9b863f3 done
Copying blob a718dac5cfe3 done
Copying blob 2ec977f639d8 done
Copying blob 959618872958 done
Copying blob 7bf88ebc3f42 done
Copying config 3d8b542d91 done
Writing manifest to image destination
Storing signatures
2021/12/02 23:16:14  info unpack layer: sha256:04c2e9b863f3ba30c9b596c91390637851a80abb18f50bb636e1ac52878d3467
2021/12/02 23:16:14  info unpack layer: sha256:a718dac5cfe3e16f73cc8316ce827ef626d5ec12dc8516ba458507eed84922d4
2021/12/02 23:16:14  info unpack layer: sha256:2ec977f639d8c2ac1535677b59ce3af5359838b2b130eafd9ecdee564349d0ee
2021/12/02 23:16:14  info unpack layer: sha256:9596188729587bdd248ed97f9efb915f413fcdcf59fdf3612bc24c4f922ac7ee
2021/12/02 23:16:15  warn rootless{usr/bin/newgidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2021/12/02 23:16:15  warn rootless{usr/bin/newuidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2021/12/02 23:16:15  warn rootless{usr/lib} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
2021/12/02 23:16:15  warn rootless{usr/lib64} ignoring (usually) harmless EPERM on setxattr "user.overlay.impure"
2021/12/02 23:16:15  warn rootless{usr/libexec/openssh/ssh-keysign} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2021/12/02 23:16:16  info unpack layer: sha256:7bf88ebc3f4276ea6cbc5fe7071460510d5fa6b78c82150e8521b588d1a81c83
INFO:    Adding owner write permission to build path: /home/adrian/tmp/build-temp-560090085/rootfs
INFO:    Creating SIF file...

but i'm curios if this is only my problem.. is the above pull working for you? the actual project is here: https://gitlab.cern.ch/asevcenc/alienpy.cont and gitlab general settings show me that is public, so the corresponding registry should work .. Thanks a lot!

[adriansev]

oh, and notice that after cleaning up the credentials, the error message is changed ..

[dtrudg]

Hmm - I'll try to play around a bit pulling from the CERN repo some more tomorrow. I can pull it without any auth from a first attempt.

04:05 PM $ singularity pull docker://gitlab-registry.cern.ch/asevcenc/alienpy.cont/alienpy:latest
INFO:    Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
Getting image source signatures
Copying blob 04c2e9b863f3 [===========>--------------------------] 11.2MiB / 35.2MiB

Are you on the cern network? Am wondering if there's some differential authentication in / out going on.

[adriansev]

no, i'm outside of cern.. and i can confirm that on lxplus also work for me...

[dtrudg]

I just had a go at this again, and it still works for me without issue, without authentication. Not able to reproduce.

Because this has sat so long, I'm going to close the issue... but if it's still a problem that you can replicate, please feel free to re-open with any relevant information.