whats-up-docker icon indicating copy to clipboard operation
whats-up-docker copied to clipboard

lscr.io failing

Open oliroe opened this issue 2 years ago • 10 comments

Hi

As of last Wednesday any of my containers running images from linuxserver are failing the check process with the error below

Anyone else having the same issue?

image

there's been no change to my setup

image

oliroe avatar Jul 01 '22 13:07 oliroe

Same issue for all lscr.io images. No changes either. Did lscr.io put some kind of authentication up ? I also have this line:

12:08:01.443 WARN whats-up-docker/watcher.docker.safeupdate: Error when processing some containers (Cannot read properties of undefined (reading 'tag'))

LeonardMeyer avatar Jul 16 '22 10:07 LeonardMeyer

I just installed WUD, and have issues with LSCR, it seems auth is required or there's a bug.

alexdelprete avatar Jul 18 '22 18:07 alexdelprete

Hi,

I confirm the issue; I'll look at it.

fmartinou avatar Jul 20 '22 21:07 fmartinou

The lscr.io API now claims for an authentication token from the Github registry 🤔

$ curl -i https://lscr.io/v2/linuxserver/radarr/tags/list

HTTP/2 401 
date: Sat, 23 Jul 2022 09:30:00 GMT
content-type: application/json
content-length: 73
www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:linuxserver/radarr:pull"
x-github-request-id: D1A2:B748:A8236E:B4E296:62DBBF98
strict-transport-security: max-age=15724800; includeSubDomains

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

It makes no sense (because the image is publicly available for pulling).

I created a new topic on the Linux Server Discourse to get support from the community. 🤞

In the mean time, I can advise you to change your containers to point to the real registry backend (ghcr.io) instead of their lscr.io proxy. Ex: ghcr.io/linuxserver/radarr:4.2.1-nightly

fmartinou avatar Jul 23 '22 09:07 fmartinou

The lscr.io API now claims for an authentication token from the Github registry

Yes that's what I discovered too when I confirmed the issue. Could we use ghcr.io token auth for lscr.io? If it's a simple proxy maybe it works...what do you think?

In the mean time, I can advise you to change your containers to point to the real registry backend (ghcr.io) instead of their lscr.io proxy.

Good advice: I did the same thing before hotio.dev was supported, I didn't think about this for lscr.io. Thanks a lot.

alexdelprete avatar Jul 23 '22 13:07 alexdelprete

@ fmartinou hey, I believe that gcr.io has the same problem ( gcr.io/cadvisor/cadvisor:v0.39.3 image ),

12:51:44.920  WARN whats-up-docker: local_cadvisor - No Registry Provider found
12:51:44.937  WARN whats-up-docker/watcher.docker.local: Error when processing (Unsupported Registry unknown) (container=local_cadvisor)
12:51:44.938  WARN whats-up-docker/watcher.docker.local: Error when processing some containers (Cannot read properties of undefined (reading 'tag'))

lukasmrtvy avatar Jul 30 '22 12:07 lukasmrtvy

I believe that gcr.io has the same problem ( gcr.io/cadvisor/cadvisor:v0.39.3 image ),

check the url, it should be ghcr.io

alexdelprete avatar Jul 30 '22 17:07 alexdelprete

  • gcr.io -> google
  • ghcr.io -> github

lukasmrtvy avatar Jul 30 '22 17:07 lukasmrtvy

Hi @lukasmrtvy ,

Your issue is a different one, I think.

For GCR, because credentials are needed, the support for this registry isn't enabled by default.

Have you configured the GCR needed properties as mentioned here?

If you haven't, please set them, remove your local state to avoid any side effects and restart the container.

fmartinou avatar Jul 30 '22 20:07 fmartinou

  • gcr.io -> google
  • ghcr.io -> github

Pardon me Lukas. Didn't even think about that...:)

alexdelprete avatar Jul 30 '22 22:07 alexdelprete

Any update about this ticket ?

Still not working on my side image

royto avatar Aug 31 '22 09:08 royto

I've just found a solution... 🥳

Because lscr.io is only a proxy in front of the Github registry, you need to provide a Github Personal Token to make it work.

That's ugly because the promise of LSCR is to provide an abstraction from the real backend used to not impact users if they decide to move to another backend 😞

As anonymous

curl -i https://lscr.io/v2/linuxserver/radarr/tags/list

HTTP/2 401 
date: Wed, 31 Aug 2022 16:52:35 GMT
content-type: application/json
content-length: 73
www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:linuxserver/radarr:pull"
x-github-request-id: A923:B619:281037:2C13CC:630F91D3
strict-transport-security: max-age=15724800; includeSubDomains

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

Providing a Github Personal Token with 'read:package' permission

curl -iH "Authorization: Bearer Zm1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxR3Q=" https://lscr.io/v2/linuxserver/radarr/tags/list

HTTP/2 200 
date: Wed, 31 Aug 2022 16:53:45 GMT
content-type: application/json
docker-distribution-api-version: registry/2.0
link: </v2/linuxserver/radarr/tags/list?last=amd64-version-3.0.0.4039&n=0>; rel="next"
x-github-request-id: 75A2:58F4:52F195:580542:630F9219
strict-transport-security: max-age=15724800; includeSubDomains

{"name":"linuxserver/radarr","tags":["amd64-v0.2.0.1504-ls81","arm32v7-v0.2.0.1504-ls81","arm64v8-v0.2.0.1504-ls81","v0.2.0.1504-ls81","amd64-v0.2.0.1504-ls34","arm32v7-v0.2.0.1504-ls34","arm64v8-v0.2.0.1504-ls34","v0.2.0.1504-ls34","amd64-3.0.0.3978-ls27","arm32v7-3.0.0.3978-ls27","arm64v8-3.0.0.3978-ls27","amd64-version-3.0.0.3978","arm32v7-version-3.0.0.3978","arm64v8-version-3.0.0.3978","3.0.0.3978-ls27","version-3.0.0.3978","amd64-3.0.0.3790-ls28","arm32v7-3.0.0.3790-ls28","arm64v8-3.0.0.3790-ls28","3.0.0.3790-ls28","amd64-3.0.0.3986-ls27","arm32v7-3.0.0.3986-ls27","arm64v8-3.0.0.3986-ls27","amd64-version-3.0.0.3986","arm32v7-version-3.0.0.3986","arm64v8-version-3.0.0.3986","3.0.0.3986-ls27","version-3.0.0.3986","amd64-3.0.0.3987-ls27","arm32v7-3.0.0.3987-ls27","arm64v8-3.0.0.3987-ls27","amd64-version-3.0.0.3987","arm32v7-version-3.0.0.3987","arm64v8-version-3.0.0.3987","3.0.0.3987-ls27","version-3.0.0.3987","amd64-v0.2.0.1504-ls35","arm32v7-v0.2.0.1504-ls35","arm64v8-v0.2.0.1504-ls35","v0.2.0.1504-ls35","amd64-3.0.0.3989-ls28","arm32v7-3.0.0.3989-ls28","arm64v8-3.0.0.3989-ls28","3.0.0.3989-ls28","amd64-3.0.0.3790-ls29","arm32v7-3.0.0.3790-ls29","arm64v8-3.0.0.3790-ls29","3.0.0.3790-ls29","amd64-3.0.0.3989-ls29","arm32v7-3.0.0.3989-ls29","arm64v8-3.0.0.3989-ls29","amd64-version-3.0.0.3989","arm32v7-version-3.0.0.3989","arm64v8-version-3.0.0.3989","3.0.0.3989-ls29","version-3.0.0.3989","amd64-v0.2.0.1504-ls82","arm32v7-v0.2.0.1504-ls82","arm64v8-v0.2.0.1504-ls82","v0.2.0.1504-ls82","amd64-3.0.0.4000-ls29","arm32v7-3.0.0.4000-ls29","arm64v8-3.0.0.4000-ls29","amd64-version-3.0.0.4000","arm32v7-version-3.0.0.4000","arm64v8-version-3.0.0.4000","3.0.0.4000-ls29","version-3.0.0.4000","amd64-3.0.0.4002-ls29","arm32v7-3.0.0.4002-ls29","arm64v8-3.0.0.4002-ls29","amd64-version-3.0.0.4002","arm32v7-version-3.0.0.4002","arm64v8-version-3.0.0.4002","3.0.0.4002-ls29","version-3.0.0.4002","amd64-3.0.0.4005-ls29","arm32v7-3.0.0.4005-ls29","arm64v8-3.0.0.4005-ls29","amd64-version-3.0.0.4005","arm32v7-version-3.0.0.4005","arm64v8-version-3.0.0.4005","3.0.0.4005-ls29","version-3.0.0.4005","amd64-v0.2.0.1504-ls36","arm32v7-v0.2.0.1504-ls36","arm64v8-v0.2.0.1504-ls36","v0.2.0.1504-ls36","amd64-3.0.0.4037-ls29","arm32v7-3.0.0.4037-ls29","arm64v8-3.0.0.4037-ls29","amd64-version-3.0.0.4037","arm32v7-version-3.0.0.4037","arm64v8-version-3.0.0.4037","3.0.0.4037-ls29","version-3.0.0.4037","amd64-3.0.0.4039-ls29","arm32v7-3.0.0.4039-ls29","arm64v8-3.0.0.4039-ls29","amd64-version-3.0.0.4039"]}

=> I'll adapt the configuration of the LSCR component so you can set a Github token

fmartinou avatar Aug 31 '22 17:08 fmartinou

When I discovered lscr.io was only a proxy to ghcr.io, I simply switched to their ghcio packages. I think that's the best solution.

alexdelprete avatar Aug 31 '22 20:08 alexdelprete

I've tried switching to ghcr.io as well but I still get HTTP 401 responses in WUD's logs. Am I missing something ? I still need to setup a token ? That's annoying...

LeonardMeyer avatar Sep 01 '22 18:09 LeonardMeyer

Please retry with a Github Personal token: see here

I'm pretty sure that LinuxServer.io team has changed the visibility of their packages, which is the root of all these troubles.

fmartinou avatar Sep 02 '22 07:09 fmartinou

I've tried switching to ghcr.io as well but I still get HTTP 401 responses in WUD's logs

Did you set GH user/token as per documentation?

image

I heavily use ghcr.io, it's much faster than docker and I don't have any heavy API limitations using authentication with GH user/token.

today at 00:00:0100:00:01.638 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/nzbget tags
today at 00:00:0100:00:01.638 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/qbittorrent tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/overseerr tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/sonarr tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/radarr tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/prowlarr tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/bazarr tags
today at 00:00:0100:00:01.639 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/readarr tags
today at 00:00:0100:00:01.640 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/plex-meta-manager tags
today at 00:00:0100:00:01.640 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/tautulli tags
today at 00:00:0100:00:01.720 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/webtop tags
today at 12:00:0012:00:00.121 DEBUG whats-up-docker/registry.ghcr: ghcr - Get linuxserver/lidarr tags

alexdelprete avatar Sep 02 '22 13:09 alexdelprete

I'm pretty sure that LinuxServer.io team has changed the visibility of their packages, which is the root of all these troubles.

I always used GH user/token to access ghcr.io. Worked perfectly since you told me it was only a GH container proxy, switched from lscr.io to ghcr.io and happy since then. :)

alexdelprete avatar Sep 02 '22 13:09 alexdelprete

Thank you for your feedback @alexdelprete :+1:

The username + token are not mandatory to configure the GHCR integration because some images can be publicly accessed.

Unfortunately, LinuxServer ones are not so you (@LeonardMeyer) have to create a token on Github and use it in your configuration.

fmartinou avatar Sep 03 '22 07:09 fmartinou

Well, it makes sense for me that if I have a user account on GH, I use it to access GHCR. I can't remember correctly, but maybe there are access API limitations also for ghcr.io...not sure, but it's better accessing it with authentication, IMHO.

Thanks for your work. :)

alexdelprete avatar Sep 03 '22 14:09 alexdelprete

In the latest 5.20.1, LSCR is not automatically enabled anymore. It's still possible to enable but username/token is now mandatory.

fmartinou avatar Sep 03 '22 16:09 fmartinou

@fmartinou There are several typos in the env var token name for LSCR in the docs https://fmartinou.github.io/whats-up-docker/#/configuration/registries/lscr/ (example, page, bottom)

It makes things a bit confusing.

LeonardMeyer avatar Sep 03 '22 17:09 LeonardMeyer

Thanks for your feedback; it's fixed (see commit here).

(waiting for Github to redeploy the static website)

fmartinou avatar Sep 03 '22 17:09 fmartinou

It's still possible to enable but username/token is now mandatory

good decision.

alexdelprete avatar Sep 03 '22 20:09 alexdelprete