aspire icon indicating copy to clipboard operation
aspire copied to clipboard

Deployment to ACS fails if you have a NPM project deployed as a container

Open shawnwildermuth opened this issue 7 months ago • 4 comments

Is there an existing issue for this?

  • [x] I have searched the existing issues

Describe the bug

When deploying to Azure Container Services from Aspire project, all deployment succeeds until the NPM project creates it's container. The error shows that it ran out of memory when contacting the Azure Container Registry (ACR). Here is the output:

2025/04/10 17:12:24 command_runner.go:310: Run exec: 'docker login --username <redacted> --password-stdin acrjcwutr6pve2ey.azurecr.io' , exit code: 1
-------------------------------------stderr-------------------------------------------
Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `Not enough memory resources are available to process this command.``

You can see the whole log (with debug output):

deployment.log

Expected Behavior

Should deploy the ACS correctly.

Steps To Reproduce

Repro case: https://github.com/shawnwildermuth/codingshorts/tree/main/aspire-deployment/before

Exceptions (if any)

N/A

.NET Version info

No response

Anything else?

Here is the output to the console:

C:\projects\codingshorts\aspire-deployment\before>azd up --debug 2> deployment.log

Packaging services (azd package)


Provisioning Azure resources (azd provision)
Provisioning Azure resources can take some time.

Subscription: 2019-MVP (5e288a25-0f05-485d-af3f-6dd01e6cb239)
Location: France Central

  (-) Skipped: Didn't find new changes.

Deploying services (azd deploy)

  (✓) Done: Deploying service cache
  - Endpoint: https://cache.internal.orangewater-20b036ce.francecentral.azurecontainerapps.io/

  (✓) Done: Deploying service db
  - Endpoint: https://db.internal.orangewater-20b036ce.francecentral.azurecontainerapps.io/

  (✓) Done: Deploying service order-processor
  - No endpoints were found

  (✓) Done: Deploying service order-queue
  - Endpoint: https://order-queue.internal.orangewater-20b036ce.francecentral.azurecontainerapps.io/

  (x) Failed: Deploying service the-store

ERROR: error executing step command 'deploy --all': failed deploying service 'the-store': failed logging into docker registry acrjcwutr6pve2ey.azurecr.io: failed logging into docker: exit code: 1, stdout: , stderr: Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `Not enough memory resources are available to process this command.``

shawnwildermuth avatar Apr 10 '25 15:04 shawnwildermuth

@vhvb1989 ?

davidfowl avatar Apr 18 '25 05:04 davidfowl

Can you try manually running docker login --username <redacted> --password-stdin acrjcwutr6pve2ey.azurecr.io

It seems like if your system is getting out of memory at the point it is trying to publish. I've seen this happen when there are several containers running at the same time as running azd up. Is that maybe the case?

vhvb1989 avatar Apr 21 '25 22:04 vhvb1989

@vhvb1989 - what credentials, and should I replace the azurecr.io with my own? A tad confused.

shawnwildermuth avatar Apr 23 '25 22:04 shawnwildermuth

@vhvb1989 - I'm still getting this. I tried with a new deployment and the docker login just hangs. When I try to use az acr login -n:

C:\projects\codingshorts\aspire-deployment\before>az acr login -n acrw5bcowmsacf5k
The output will be changed in next breaking change release(2.73.0) scheduled for May 2025. Exit code will be 1 if command fails for docker login.
Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `Not enough memory resources are available to process this command.``
Login failed.

NOTE that this happened on a .NET project this time (before it got to the NPM project), so I don't think it's specific to that. I can try to use an existing ACR, but since I'm trying to show how this works, I'd rather not workaround but understand the issue.

Note: where who is running out of memory. I have 64 gig on this machine and nowhere close to running out of memory.

Some machine info if it helps:

machine.zip

Also, I've tried this for US and European data centers (I'm in Europe if that helps)

shawnwildermuth avatar Apr 27 '25 15:04 shawnwildermuth

Any news or hints? Our should I open s support ticket?

shawnwildermuth avatar May 01 '25 14:05 shawnwildermuth

Hey @shawnwildermuth , this could be either a Docker issue or a Windows Credentials Manager issue.

Have you tried this:

  • Manually Remove Credentials:
    • Open Control Panel → User Accounts → Manage Windows Credentials.
    • Navigate to Windows Credentials and delete unnecessary entries.

OR

  • Increase Docker Memory Allocation:
    • Open Docker Desktop → Settings → Resources → Advanced.
    • Increase the memory allocation and restart Docker.

vhvb1989 avatar May 01 '25 16:05 vhvb1989

@vhvb1989

I tried to increase the Docker Memory Allocation but it says that WSL2 defines it. And since I don't have a .wslconfig file, I am not sure what to change.

shawnwildermuth avatar May 02 '25 18:05 shawnwildermuth

@shawnwildermuth did you get this working?

davidfowl avatar Jun 03 '25 06:06 davidfowl

Yes, we can close this. Eventually the service returned to working state. removing the credentials and re-authenticating seemed to do the trick.

shawnwildermuth avatar Jun 03 '25 12:06 shawnwildermuth