ion icon indicating copy to clipboard operation
ion copied to clipboard
verified publisher icon sst

Slow deployments (15-20 mins+) with `sst deploy`

Open apetta opened this issue 1 year ago • 0 comments

Currently on sst 3.1.49.

Deployment times have increased a lot recently. Something is causing the deployment to hang for a long time (10 mins+) between some steps. Have tried --verbose and it doesn't give too many clues.

Here's a slice of the logs from a recent deployment:

time=2024-09-30T12:57:50.214+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.291+01:00 level=INFO msg="found existing bootstrap" data="{\"version\":4,\"asset\":\"sst-asset-bnmoxdeoenmo\",\"assetEcrRegistryId\":\"XXXXX\",\"assetEcrUrl\":\"XXXXX.dkr.ecr.eu-west-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-avassxxbcsca\"}"
time=2024-09-30T12:57:50.410+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:57:50.418+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.418+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:57:50.421+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.422+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.422+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.423+01:00 level=INFO msg="found existing bootstrap" data="{\"version\":4,\"asset\":\"sst-asset-bnmoxdeoenmo\",\"assetEcrRegistryId\":\"XXXXX\",\"assetEcrUrl\":\"XXXXX.dkr.ecr.eu-west-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-avassxxbcsca\"}"
time=2024-09-30T12:57:50.423+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.425+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.426+01:00 level=INFO msg="found existing bootstrap" data="{\"version\":4,\"asset\":\"sst-asset-bnmoxdeoenmo\",\"assetEcrRegistryId\":\"XXXXX\",\"assetEcrUrl\":\"XXXXX.dkr.ecr.eu-west-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-avassxxbcsca\"}"
time=2024-09-30T12:57:50.426+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.426+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.427+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.433+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T12:57:50.433+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:57:50.439+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.967+01:00 level=INFO msg="rpc request" method=POST url=/rpc
time=2024-09-30T12:57:50.968+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.970+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.970+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:50.980+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T12:57:54.586+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:57:54.590+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T12:57:54.591+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:57:54.594+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:54.594+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:57:54.637+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T12:57:55.356+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:58:00.778+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:58:00.788+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:58:00.788+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T12:58:00.794+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T12:58:00.794+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T12:58:00.800+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.477+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.480+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.480+01:00 level=INFO msg="rpc request" method=POST url=/rpc
time=2024-09-30T13:11:54.483+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.515+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:11:54.516+01:00 level=INFO msg="rpc request" method=POST url=/rpc
time=2024-09-30T13:11:54.518+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.520+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.520+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:11:54.545+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:02.667+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:02.674+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:02.675+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:02.678+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:02.685+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:02.686+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:02.686+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:03.153+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:03.161+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:03.161+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:03.164+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:03.172+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:03.172+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:03.173+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:05.088+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:05.184+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:05.316+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:05.451+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.618+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:10.627+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.628+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.628+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:10.634+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.635+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.635+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.896+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.944+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:10.952+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.953+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.953+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:10.953+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:10.955+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:10.959+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:11.028+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:11.109+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:11.202+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:11.208+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:11.375+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:11.383+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:11.390+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:24.842+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:24.849+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:25.703+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:26.356+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent
time=2024-09-30T13:12:26.587+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:26.592+01:00 level=INFO msg=publishing type=*apitype.ResourcePreEvent
time=2024-09-30T13:12:26.601+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:26.606+01:00 level=INFO msg=publishing type=*apitype.ResOutputsEvent
time=2024-09-30T13:12:26.606+01:00 level=INFO msg=publishing type=*apitype.SummaryEvent
time=2024-09-30T13:12:26.611+01:00 level=INFO msg=publishing type=*apitype.CancelEvent
time=2024-09-30T13:12:27.335+01:00 level=INFO msg="done running stack command"

There's a really long delay at time=2024-09-30T12:58:00.800+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent time=2024-09-30T13:11:54.477+01:00 level=INFO msg=publishing type=*apitype.DiagnosticEvent

It also sometime errors out after really long wait times saying the folder/file for handlers can't be found. Other times, it deploys successfully after the really long delays.

Deployment times logged by the CLI in the terminal all seem 'reasonable' though: image

apetta avatar Sep 30 '24 12:09 apetta