[Bug?]: "Libzip Error: Malloc failure" when installing certain packages

[MLoughry]

Self-service

• [X] I'd be willing to implement a fix

Describe the bug

I just moved my team from v1 to v4.02 this past week. Unfortunately, when trying to upgrade one of our Microsoft-internal packages, we ran into the following error:

➤ YN0001: │ Libzip Error: Malloc failure
    at ZipFS.makeLibzipError ([worker eval]:1:296093)
    at ZipFS.getBufferAndClose ([worker eval]:1:296860)
    at ZipFS.saveAndClose ([worker eval]:1:298325)
    at convertToZipWorker ([worker eval]:1:407245)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MessagePort.<anonymous> ([worker eval]:1:409017)

I have traced the line throwing the error, but I'm not familiar enough with the code or libzip to debug further.

This seems to have regressed between 3.7.0 and 4.0.0, because the former installs the package without issues. My suspicion is that it's related to #4247.

To reproduce

Unfortunately, the package is Microsoft internal and cannot be freely shared. I have reached out to the owners of the package to see if we can share it out for this repro (potentially with an NDA), and will update when I have more information.

Environment

System:
    OS: Linux 6.2 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    CPU: (32) x64 AMD EPYC 7763 64-Core Processor
  Binaries:
    Node: 18.18.0 - /tmp/xfs-6327a856/node
    Yarn: 4.0.0 - /tmp/xfs-6327a856/yarn
    npm: 9.8.1 - /usr/local/share/nvm/versions/node/v18.18.0/bin/npm

Additional context

No response

[arcanis]

"Malloc failure" would suggest a memory issue. Is it a very large package ?

[MLoughry]

The zip archive is about 4MB, and unzipped it’s about 6MB.

[MLoughry]

Ok, scratch that last comment. Another dev pointed out it could be one of the package’s dependencies, and it turns out that it was: a localized strings package that is 245MB zipped and 1.2GB unzipped.

I'm going to go browbeat whoever thought that was okay, but the fact remains that this is still a regression from 3.7.0 ☹️

[merceyz]

Could you try with compressionLevel: 0 in .yarnrc.yml?

If you're willing could you also test https://github.com/yarnpkg/berry/pull/6011 with compressionLevel set to whatever it was originally?

yarn set version from sources --branch 6011

[MLoughry]

We were already using compressionLevel: 0

compressionLevel: 0

enableGlobalCache: true

nodeLinker: node-modules

npmAlwaysAuth: true

npmRegistryServer: <Internal Azure DevOps Feed>

yarnPath: .yarn/releases/yarn-4.0.2.cjs

Unfortunately, the changes in #6011 did not resolve the issue.

[Keisuke69xx]

I have the same issue. Using compressionLevel:0 and #6011 didn't resolve this issue for us as well.

[Archish27]

Any updates on this? Please someone help