can't push big size image in harbor

[AndrewSindov]

Hello! Unable to upload a large image to the harbor. There are no problems with images smaller than 1GB. I looked through all the logs and this is what I found: "Upload failed, retrying: received unexpected HTTP status: 524 " - docker "http: proxy error: context canceled" - core.log "SSL_read: 9 SSL_read: -1 SSL_get_error: 2" - debug error.log from nginx

I tried to locally deploy a harabor and send large images to it, everything works fine. The harbor domain is behind cloudflare and nginx. Suspecting that the problem is related to them, but so far I can not find out for sure. Maybe you can tell me. Below is my nginx config.

upstream core {
  server 127.0.0.1:8280;
}

upstream portal {
  server 127.0.0.1:8180;
}

server {
#  error_log /var/log/harbor/debug.log debug;
  listen 443 ssl;
  server_name example.com;
  server_tokens off;
  # SSL
    ssl_certificate /path/;
    ssl_certificate_key /path/;

  # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers '!aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES:';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;


  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  # Add extra headers
  add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
  add_header X-Frame-Options DENY;
  add_header Content-Security-Policy "frame-ancestors 'none'";

  # costumized location config file can place to /etc/nginx dir with prefix harbor.https. and suffix .conf
  include /etc/nginx/conf.d/harbor.https.*.conf;

  location / {
    proxy_pass http://portal/;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_cookie_path / "/; HttpOnly; Secure";

    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /c/ {
    proxy_pass http://core/c/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_cookie_path / "/; Secure";

    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /api/ {
    proxy_pass http://core/api/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_cookie_path / "/; Secure";

    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /chartrepo/ {
    proxy_pass http://core/chartrepo/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_cookie_path / "/; Secure";

    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /v1/ {
    return 404;
  }

  location /v2/ {
    proxy_pass http://core/v2/;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /service/ {
    proxy_pass http://core/service/;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_cookie_path / "/; Secure";

    proxy_buffering off;
    proxy_request_buffering off;

  }

  location /service/notifications {
    return 404;
  }

}

server {
    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name example.com;
    return 308 https://$host/$request_uri;


}

[chlins]

How long time when you encountered this issue while pushing the image?

[AndrewSindov]

How long time when you encountered this issue while pushing the image?

What do you mean? Yesterday I ran into this problem.

[chlins]

Could you post the output of $ time docker push {your-image}?

[AndrewSindov]

Could you post the output of $ time docker push {your-image}?

After an unsuccessful push, retry begins. I stopped the command and got this time:

real	2m57.693s
user	0m0.095s
sys	0m0.069s

[chlins]

@AndrewSindov Could you refer to this https://support.cloudflare.com/hc/en-us/articles/115003011431-Error-524-A-timeout-occurred#524error and debug by adjust the timeout?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.