Problem configuring proxy for docker.io

[piontec]

Hey! I've configured containerd as below:

root@wc1-oci-reg-control-plane:/etc/containerd/certs.d# cat docker.io/hosts.toml 
server = "https://registry-1.docker.io"

[host."http://172.17.0.1:30180"]
	capabilities = ["pull", "resolve"]

Proxy seems to be deployed fine:

root@wc1-oci-reg-control-plane:/etc/containerd/certs.d# curl -s 172.17.0.1:30180/metrics | head
# HELP http_incoming_requests Incoming Requests
# TYPE http_incoming_requests counter
http_incoming_requests{endpoint="/",method="GET",status="200"} 206
http_incoming_requests{endpoint="/metrics",method="GET",status="200"} 4
http_incoming_requests{endpoint="/v2/{image:[^{}]+}/manifests/{reference}",method="HEAD",status="500"} 1
# HELP http_response_code Response Codes

Still, when I try to pull a simple image from docker hub:

root@wc1-oci-reg-control-plane:/etc/containerd/certs.d# crictl pull docker.io/alpine:latest
Image is up to date for sha256:7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb

I get some 403 in the oci-registry log stream and nothing is saved in the data directory:

Logs

2023-09-28T13:44:05.762002Z DEBUG reqwest::connect: starting new connection: https://docker.io/    
2023-09-28T13:44:05.762247Z DEBUG hyper::client::connect::dns: resolving host="docker.io"
2023-09-28T13:44:11.774785Z DEBUG hyper::client::connect::http: connecting to 54.165.156.197:443
2023-09-28T13:44:11.887714Z DEBUG hyper::client::connect::http: connected to 54.165.156.197:443
2023-09-28T13:44:11.887741Z DEBUG rustls::client::hs: No cached session for DnsName("docker.io")    
2023-09-28T13:44:11.887790Z DEBUG rustls::client::hs: Not resuming any session    
2023-09-28T13:44:12.000008Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256    
2023-09-28T13:44:12.000046Z DEBUG rustls::client::tls13: Not resuming    
2023-09-28T13:44:12.000356Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck]    
2023-09-28T13:44:12.000366Z DEBUG rustls::client::hs: ALPN protocol is None    
2023-09-28T13:44:12.000914Z DEBUG hyper::proto::h1::io: flushed 378 bytes
2023-09-28T13:44:12.338080Z DEBUG hyper::proto::h1::io: parsed 2 headers
2023-09-28T13:44:12.338106Z DEBUG hyper::proto::h1::conn: incoming body is empty
2023-09-28T13:44:12.338161Z DEBUG hyper::client::pool: pooling idle connection for ("https", docker.io)
2023-09-28T13:44:12.338206Z DEBUG reqwest::async_impl::client: redirecting 'https://docker.io/v2/library/alpine/manifests/latest?ns=docker.io' to 'https://www.docker.com/v2/library/alpine/manifests/latest?ns=docker.io'    
2023-09-28T13:44:12.338238Z DEBUG hyper::client::pool: reuse idle connection for ("https", www.docker.com)
2023-09-28T13:44:12.338309Z DEBUG Connection: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(3), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
2023-09-28T13:44:12.353054Z DEBUG Connection: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(3), flags: (0x4: END_HEADERS) } peer=Client
2023-09-28T13:44:12.353139Z DEBUG Connection: h2::codec::framed_read: received frame=Data { stream_id: StreamId(3) } peer=Client
2023-09-28T13:44:12.353152Z DEBUG Connection: h2::codec::framed_read: received frame=Data { stream_id: StreamId(3), flags: (0x1: END_STREAM) } peer=Client
2023-09-28T13:44:12.353282Z DEBUG hyper::client::pool: reuse idle connection for ("https", docker.io)
2023-09-28T13:44:12.353414Z DEBUG hyper::proto::h1::io: flushed 365 bytes
2023-09-28T13:44:12.465971Z DEBUG hyper::proto::h1::io: parsed 2 headers
2023-09-28T13:44:12.465987Z DEBUG hyper::proto::h1::conn: incoming body is empty
2023-09-28T13:44:12.466013Z DEBUG hyper::client::pool: pooling idle connection for ("https", docker.io)
2023-09-28T13:44:12.466036Z DEBUG reqwest::async_impl::client: redirecting 'https://docker.io/v2/library/alpine/manifests/latest' to 'https://www.docker.com/v2/library/alpine/manifests/latest'    
2023-09-28T13:44:12.466054Z DEBUG hyper::client::pool: reuse idle connection for ("https", www.docker.com)
2023-09-28T13:44:12.466092Z DEBUG Connection: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(5), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
2023-09-28T13:44:12.489660Z DEBUG Connection: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(5), flags: (0x4: END_HEADERS) } peer=Client
2023-09-28T13:44:12.489703Z DEBUG Connection: h2::codec::framed_read: received frame=Data { stream_id: StreamId(5) } peer=Client
2023-09-28T13:44:12.489711Z DEBUG Connection: h2::codec::framed_read: received frame=Data { stream_id: StreamId(5), flags: (0x1: END_STREAM) } peer=Client
2023-09-28T13:44:12.489757Z ERROR oci_registry::api::error: 500: Error with upstream registry: unexpected HTTP status 403 Forbidden
2023-09-28T13:44:12.489802Z DEBUG actix_web::middleware::logger: Error in response: Upstream(UnexpectedHttpStatus(403))    
2023-09-28T13:44:12.489829Z  INFO actix_web::middleware::logger: 10.244.0.1 "HEAD /v2/library/alpine/manifests/latest?ns=docker.io HTTP/1.1" 500 66 "-" "containerd/v1.7.1" 19.567011    
2023-09-28T13:44:17.571069Z DEBUG rustls::common_state: Sending warning alert CloseNotify

Tested on helm chart v0.3.20 with the following config file

replicas: 1
registry:
  storage:
    mode: filesystem
service:
  type: NodePort