Recreate: Request failed with status code 504

[keremenci]

Before you start please confirm the following.

• [X] Yes, I've searched similar issues on GitHub.
• [X] Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

Whenever I try to recreate some of my containers I get Failure: Request failed with status code 504

The containers in question are relatively large, around 10G. I can manually pull the images, the registry seems fine.

Expected Behavior

The container is recreated and changes in the image are applied

Actual Behavior

Recreation process takes a relatively long time and ends with the message: "Request failed with status code 504". Changes are not applied.

Steps to Reproduce

1. Build a relatively big docker image (~5G)
2. Push to a self-hosted docker registry (mine is also hosted on portainer)
3. Create a container and run it
4. Make some changes, rebuild and push the docker image
5. Recreate the container to apply changes
6. Wait a bit
7. Error

Portainer logs or screenshots

Portainer logs:

2024/02/05 10:38AM DBG github.com/portainer/portainer/api/docker/container.go:51 > starting to parse image | image=<self-hosted-registry>/<image>:latest
2024/02/05 10:38AM DBG github.com/portainer/portainer/api/docker/images/puller.go:29 > starting to pull the image | image=<self-hosted-registry>/<image>:latest
2024/02/05 10:39AM DBG github.com/portainer/portainer/api/http/security/bouncer.go:250 > HTTP error | error="pull image error <self-hosted-registry>/<image>:latest: context canceled" msg="Error recreating container" stack_trace=[{"func":"(*ContainerService).Recreate","line":"75","source":"container.go"},{"func":"(*Handler).recreate","line":"50","source":"recreate.go"},{"func":"LoggerHandler.ServeHTTP","line":"23","source":"error.go"},{"func":"(*RequestBouncer).mwUpgradeToRestrictedRequest.func1","line":"250","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*RequestBouncer).mwAuthenticateFirst.func1","line":"301","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"mwSecureHeaders.func1","line":"401","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Router).ServeHTTP","line":"210","source":"mux.go"},{"func":"dockerOnlyMiddleware.func1","line":"64","source":"handler.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"WithEndpoint.func1.1","line":"46","source":"endpoint.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Router).ServeHTTP","line":"210","source":"mux.go"},{"func":"StripPrefix.func1","line":"2165","source":"server.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Handler).ServeHTTP","line":"201","source":"handler.go"},{"func":"(*OfflineGate).WaitingMiddleware.func1","line":"48","source":"offlinegate.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Monitor).WithRedirect.func1","line":"118","source":"admin_monitor.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"WithSlowRequestsLogger.func1","line":"26","source":"slow_request_logger.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"serverHandler.ServeHTTP","line":"2936","source":"server.go"},{"func":"(*conn).serve","line":"1995","source":"server.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}] status_code=500
2024/02/05 10:39AM DBG github.com/portainer/portainer/api/http/middlewares/slow_request_logger.go:33 > slow request | elapsed_ms=96454.35138 method=POST url=/api/docker/2/containers/eff4f4ff94d852f87ece2881c12e73bcfcaa8863ee834528704adc3f7b108e14/recreate
2024/02/05 10:39AM DBG github.com/portainer/portainer/api/docker/container.go:44 > starting to fetch container information | container_id=eff4f4ff94d852f87ece2881c12e73bcfcaa8863ee834528704adc3f7b108e14
2024/02/05 10:39AM DBG github.com/portainer/portainer/api/docker/container.go:51 > starting to parse image | image=<self-hosted-registry>/<image>:latest
2024/02/05 10:39AM DBG github.com/portainer/portainer/api/docker/images/puller.go:29 > starting to pull the image | image=<self-hosted-registry>/<image>:latest
2024/02/05 10:40AM DBG github.com/portainer/portainer/api/http/middlewares/slow_request_logger.go:33 > slow request | elapsed_ms=334.119521 method=GET url=/api/system/version
2024/02/05 10:41AM DBG github.com/portainer/portainer/api/http/security/bouncer.go:250 > HTTP error | error="pull image error <self-hosted-registry>/<image>:latest: context canceled" msg="Error recreating container" stack_trace=[{"func":"(*ContainerService).Recreate","line":"75","source":"container.go"},{"func":"(*Handler).recreate","line":"50","source":"recreate.go"},{"func":"LoggerHandler.ServeHTTP","line":"23","source":"error.go"},{"func":"(*RequestBouncer).mwUpgradeToRestrictedRequest.func1","line":"250","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*RequestBouncer).mwAuthenticateFirst.func1","line":"301","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"mwSecureHeaders.func1","line":"401","source":"bouncer.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Router).ServeHTTP","line":"210","source":"mux.go"},{"func":"dockerOnlyMiddleware.func1","line":"64","source":"handler.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"WithEndpoint.func1.1","line":"46","source":"endpoint.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Router).ServeHTTP","line":"210","source":"mux.go"},{"func":"StripPrefix.func1","line":"2165","source":"server.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Handler).ServeHTTP","line":"201","source":"handler.go"},{"func":"(*OfflineGate).WaitingMiddleware.func1","line":"48","source":"offlinegate.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"(*Monitor).WithRedirect.func1","line":"118","source":"admin_monitor.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"WithSlowRequestsLogger.func1","line":"26","source":"slow_request_logger.go"},{"func":"HandlerFunc.ServeHTTP","line":"2122","source":"server.go"},{"func":"serverHandler.ServeHTTP","line":"2936","source":"server.go"},{"func":"(*conn).serve","line":"1995","source":"server.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}] status_code=500
2024/02/05 10:41AM DBG github.com/portainer/portainer/api/http/middlewares/slow_request_logger.go:33 > slow request | elapsed_ms=134.117337 method=GET url=/api/system/version

Portainer version

2.19.4

Portainer Edition

Community Edition (CE)

Platform and Version

Docker 24.0.4

OS and Architecture

Ubuntu 22.04 AMD64

Browser

Opera 106.0.4998.66

What command did you use to deploy Portainer?

: ai-portainer
services:
    portainer-ce:
        ports:
            - 8000:8000
            - 9443:9443
        container_name: portainer
        restart: always
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - portainer_data:/data
        image: portainer/portainer-ce:latest
volumes:
    portainer_data:
        name: portainer_data
        external: true

Additional Information

Trying the above steps in portainer-ce 2.18.4 works as expected.

[jonrick]

I'm having the same issue.

[JCPvanHooren]

Seems like I'm having the same issue here. I can work around it by manually removing and pulling the image using docker cli, so it seems the issue is with re-pulling the image.

[tobsynl]

Yep, same issue.

[ChrispyPeaches]

I'm having the same issue

[ChrispyPeaches]

The problem was solved for me by enabling "Skip TLS Verification"

[orangedmind]

@ChrispyPeaches How did you enable "Skip TLS Verification"?

[NetRat88]

Have the same issue. Would also be interessted where I can enable "Skip TLS Verification"?

[ChrispyPeaches]

@orangedmind @NetRat88

In the stack details page, when selecting "Advanced Configuration", the option appears for my stack.

• Note that I'm using Portainer Business Edition (you can get a free licence for personal use)

Hope this helps!

Stack Details Page

Advanced Configuration Dropdown

[vlec-be]

Same issue for me except I don't use stacks for the containers I want to recreate (example: Home Assistant)

[JCPvanHooren]

My workaround is to go through the steps manually:

1. Stop the stack/specific container
2. Re-pull the image
3. Update the stack to make it restart the container

Seems like it's related to taking too long to re-pull the image.