Reduce number of system calls made during yarn install

[trivikr]

Discussed in https://github.com/yarnpkg/berry/discussions/4023

^{Originally posted by trivikr January 26, 2022} The blog post on In-depth of tnpm rapid mode.

In this blog post the author sees [email protected] making ~4X syscalls as compared to pnpm/tnpm as shown in the screenshot below.

Screenshot

Related twitter thread: https://twitter.com/arcanis/status/1486059029235388418

• [ ] I'd be willing to implement this feature (contributing guide)
• [x] This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

As a customer of yarn, I do not want yarn to make significantly more number of system calls as compared to other package managers.

When tested with node_modules + central store mode, [email protected] makes ~3X syscalls as compared to pnpm. This is discussed prior in https://github.com/yarnpkg/berry/discussions/4023, and on Discord thread.

System information

$ hostnamectl
...
  Operating System: Amazon Linux 2
       CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2:-:internal
            Kernel: Linux 5.4.172-100.336.amzn2int.x86_64
      Architecture: x86-64

$ node -v
v16.13.2

$ npm init [email protected]
✔ Project name: … vite-project
✔ Select a framework: › vue
✔ Select a variant: › vue-ts

$ cd vite-project && git init && git add. && git commit -am "Initial commit"

yarn v3.1.1: cold - 221068, warm - 37702

$ git clean -dfx && git checkout .

$ yarn set version stable
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.1.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.1.1.cjs
➤ YN0000: Done in 0s 353ms

$ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml

$ yarn --version
3.1.1

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00   35.051901                221068     13692 total
strace -c -f yarn  4.33s user 3.58s system 147% cpu 5.354 total

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00    5.470758                 37702      3699 total
strace -c -f yarn  1.26s user 0.62s system 152% cpu 1.235 total

pnpm v6.29.1: cold - 74734, warm - 28430

$ git clean -dfx && git checkout .

$ echo 'node-linker=hoisted' > .npmrc

$ pnpm --version
6.29.1

$ time strace -c -f pnpm install
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00    4.355469                 74734      8987 total
strace -c -f pnpm install  1.75s user 1.03s system 117% cpu 2.368 total

$ time strace -c -f pnpm install
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00    1.895735                 28430      1936 total
strace -c -f pnpm install  1.08s user 0.44s system 129% cpu 1.173 total

Describe the solution you'd like

yarn to make less number of system calls while running install.

Describe the drawbacks of your solution

N/A

Describe alternatives you've considered

N/A

[trivikr]

Refs: https://github.com/yarnpkg/berry/pull/4036

[trivikr]

Verifying https://github.com/yarnpkg/berry/pull/4036, and several other perf fixes in v3.2.0-rc.14

yarn v3.2.0-rc.14: cold - 283933, warm - 37379

$ git clean -dfx && git checkout .

$ yarn set version 3.2.0-rc.14    
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.0-rc.14/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.0-rc.14.cjs
➤ YN0000: Done in 0s 424ms

$ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml

$ yarn --version
3.2.0-rc.14

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00   85.094174                283933     23559 total
strace -c -f yarn  7.56s user 5.74s system 149% cpu 8.912 total

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00    5.029950                 37379      3765 total
strace -c -f yarn  1.26s user 0.61s system 153% cpu 1.218 total

[merceyz]

#4036 shouldn't affect the node-modules linker since it doesn't use it.

[trivikr]

Verifying https://github.com/yarnpkg/berry/pull/4101, published in v3.2.0-rc.16

yarn v3.2.0-rc.16: cold - 211006, warm - 35565

$ git clean -dfx && git checkout .

$ yarn set version 3.2.0-rc.16  
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.0-rc.16/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.0-rc.16.cjs
➤ YN0000: Done in 0s 790ms

$ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml

$ yarn --version
3.2.0-rc.16

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00   35.467709                211006     17482 total
strace -c -f yarn  4.06s user 3.57s system 147% cpu 5.176 total

$ time strace -c -f yarn
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
...
------ ----------- ----------- --------- --------- ----------------
100.00    7.148630                 35565      4057 total
strace -c -f yarn  1.11s user 0.71s system 152% cpu 1.191 total