Cannot resolve layer for ghcr.io

[itsmurugappan]

Hi Team,

I get the below error while pulling the images from ghcr. The images is a estgz image.

snapshotter-version := HEAD

{"error":"cannot resolve layer: failed to redirect (host \"ghcr.io\", ref:\"ghcr.io/itsmurugappan/java-hw-estgz:v1\", digest:\"sha256:22ba525eead2839b02eebd3f096b957997c3935e8744f5fb46c3a9a987c9488c\"): failed to request: failed to fetch oauth token: unexpected status: 403 Forbidden: failed to resolve","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/56/fs","msg":"failed to resolve source","src":"ghcr.io/itsmurugappan/java-hw-estgz:v1/sha256:22ba525eead2839b02eebd3f096b957997c3935e8744f5fb46c3a9a987c9488c","time":"2020-12-20T01:38:18.685688065Z"}

Not sure, why I get 403, the image is public and am able to pull it

[ktock]

@itsmurugappan Thanks for reporting this. Does this occur everytime the snapshotter pulls ghcr.io/itsmurugappan/java-hw-estgz:v1 ?

[itsmurugappan]

Happens for other images on ghcr also

{"error":"failed to resolve layer \"sha256:8de7d11f0e064ec92e36a7d4b319c3fc2c6b6b9b085c3a74def59196f7e5d401\" from \"ghcr.io/itsmurugappan/go-hw-estgz:v1\": failed to resolve layer: failed to resolve the source: cannot resolve layer: failed to redirect (host \"ghcr.io\", ref:\"ghcr.io/itsmurugappan/go-hw-estgz:v1\", digest:\"sha256:8de7d11f0e064ec92e36a7d4b319c3fc2c6b6b9b085c3a74def59196f7e5d401\"): failed to request: failed to fetch oauth token: unexpected status: 403 Forbidden: failed to resolve: failed to resolve target","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/11/fs","msg":"failed to resolve layer","time":"2020-12-22T17:40:02.595337952Z"}
Dec 22 17:40:02 rpi-master containerd-stargz-grpc[577551]: {"error":null,"key":"k8s.io/20/extract-492102989-7ij5 sha256:2950acc5fb49572399c03c8505ea046ac01f4d3982c02081938f7c2297cb75bd","level":"debug","msg":"failed to prepare remote snapshot","parent":"k8s.io/19/sha256:d92c5478a5efd866cea6be3c0c14a08af5a93d38ca73d6974a756af54c43ec7a","remote-snapshot-prepared":"false","time":"2020-12-22T17:40:02.595503561Z"}

Interestingly this happens only on one node. Not sure what i am missing

[mattmoor]

I may have just opened a duplicate of this: https://github.com/containerd/stargz-snapshotter/issues/227

[PKizzle]

Also happens for AWS ECR registries:

{"error":"failed to refresh URL on status 403 Forbidden", ...}
{"error":"cannot resolve layer: failed to redirect (host \"XXXXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com\", ...}

Edit: This seems to be an authentication bug. After the first successful authentication it is possible to download image layers. However, once the authentication token expires it is never refreshed and causes layer pulling issues. My current workaround is to use a cronjob that re-authenticates every few hours.

[maxpain]

The same problem with Google Cloud Artifact Registry