Improve reliability of `bun install`

[Jarred-Sumner]

Reliability:

• [x] The package hoister needs to correctly handle when adjacent packages depend on conflicting versions of a package after one dependency has already been hoisted to a parent. These packages are resolved, downloaded, and extracted. They're just not being installed in the right place 100% of the time.
• [x] There are infrequent and difficult to reproduce data corruption bugs. I think a simple checksum like CRC32 for the manifest file and the lockfile would be a good idea. However, it might be something more trivial related to object pooling. That answer seems to better fit the pattern
• [x] In certain cases, the https client hangs in the TLS state machine. I have not seen this with npm, but I can consistently reproduce it with https://lodash.com.
• [ ] bun remove should remove nested dependencies rather than literally only what is entered in the CLI

Missing features:

• [x] git: packages
• [x] github: packages
• [x] link: packages & workspaces
• [ ] node-gyp support (detect binding.gyp). This may involve tweaking the lockfile format to store which packages need node-gyp / a very limited form of postinstall
• [ ] Tarball packages

Behavior tests

These need lots of variations and need to run on each platform + os pair.

• [ ] Lockfile invalidation – verify a lockfile doesn't change when there are no "real" changes. Verify that lockfiles between OS/CPU are consistent (since we are relying on the C ABI and constraining ourselves to 64 bit CPUs, it should never be inconsistent, however test coverage > "it should never")
• [ ] require tests – run bun's node_modules resolver on a resolved package tree and assert the versions match all dependencies
• [ ] Poor network connectivty – tests that ensure packages eventually install correctly under low quality network connections. This may be a test that should run daily/hourly rather than on every push

[Jarred-Sumner]

I believe the data corruption was caused by the HTTP client sometimes closing file descriptors (for sockets) multiple times. That, along with the TLS handshaking issue was fixed in Bun v0.0.69.

[Jarred-Sumner]

Bugs fixed today:

• When linking one bin failed, it would stop linking the rest in the package. Now it doesn't - 95c0f28deef5d78f11177caba0c973a431a5a312
• There was an edgecase where bun install wouldn't set permissions correctly when linking 185f74dfae6a1c755915939fe4b7d69cad3f03af
• When extracting tarballs, bun previously skipped symlinks, empty files, and empty directories. Now it doesn't - 8fca3f24879be7eeea35f6e43c2481c502f12aff. Skipping directories & empty files is unsafe because code may expect a file/folder to exist. I do not believe npm supports symlinks in packages, but the code should be there just in case
• There was a bug where extracting tarballs with long file names would fail 90189866008a6dd8726ef94d4d795c942bf01339
• Linking bins from scoped packages didn't work ce442aa3e13195df7f232dac992dc84e5a933ac5

[Jarred-Sumner]

Today's changes:

• [bun install] Make global bin dir configurable 18b1a36d84593ea71bdae9f42d175b96eb7e99d4
• [bun install] Print linked bin names and improve output 89c99700f664cb4e32f9af1bc4b18ef5064129da
• [bun install] bun pm bin prints the bin directory 7c6386d81e2233dadd4816bb3e3029ec9befaaa8
• Add bun r as an alias for bun remove 757a1686f07a04c17b1d9384839a6b1a229b9989
• Add bun -g to completions 611cdd69b1d714f7365e0a80f0a7769a5ae4ef83
• [bun install] Implement global installs 90c573cd42be1abab86db5820d1f6d1666eade8e
• [bun install] Implement private registry support & scoped packages 8570b4a9d7a4acd7e528912c5d6a9f71a63f5c9b
• [bun install] Implement bunfig.toml config d67c95d8ebb366d14976ce74e9448a23c3d6886a
• [bun install] Fix bun add multiple packages writing malformed package.json fce94e29a00f7b0b533c2725fe36b807a61ea517

[ArnaudBarre]

In missing features I would add running postinstall script (from the main package.json)

In edge cases: Removing a dependency from the package json manually doesn't remove anything when running bun again (even with new dependencies)

[Jarred-Sumner]

Yeah, sounds reasonable to do pre/post scripts for the main project

On Fri, Feb 25, 2022 at 5:01 PM Arnaud Barré @.***> wrote:

In missing feature I would add running postinstall script (from the main package.json)

— Reply to this email directly, view it on GitHub https://github.com/Jarred-Sumner/bun/issues/101#issuecomment-1051408201, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFNGS5WUELQEPE2XJDUY2LU5AQ5XANCNFSM5LHIVIXA . You are receiving this because you authored the thread.Message ID: @.***>

[FinnRG]

npm: dependencies are also not fully supported. (#227)

[SheetJSDev]

There are two interesting (and currently unsupported) cases with "tarballs":

1. specifying a URL for the tarball:

cat <<EOF >package.json
{"dependencies":{"xlsx":"https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz"}}
EOF
bun i

(error: xlsx@https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz failed to resolve)

2. manually downloading tarball and specifying a file: url in package.json:

curl -LO https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz
cat <<EOF >package.json
{"dependencies":{"xlsx":"file:./xlsx-latest.tgz"}}
EOF
bun i

(error: xlsx@file:./xlsx-latest.tgz failed to resolve)

[manast]

Postinstall FTW. Some packages like Puppeteer does not seem to work without it:

Could not find Chromium (rev. 1083080). This can occur if either
  1. you did not perform an installation before running the script (e.g. `npm install`) or
  2. your cache path is incorrectly configured (which is: /Users/manuelastudillo/.cache/puppeteer).
 For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration. (via Error)

 Original error stack trace: Error: Could not find Chromium (rev. 1083080). This can occur if either
  1. you did not perform an installation before running the script (e.g. `npm install`) or
  2. your cache path is incorrectly configured (which is: /Users/manuelastudillo/.cache/puppeteer).
 For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.

[wacdev]

bun i not work with link:../_ in pacakge.json as image above

[xxleyi]

on macOS 13.2.1, withbun i on a large react project, has a weird error:

GET https://api.github.com/repos/mrmlnc/readdir-enhanced/tarball/ISSUE-11_monkey_fix - 404
error: readdir-enhanced@github:mrmlnc/readdir-enhanced#ISSUE-11_monkey_fix failed to resolve

but bun add @mrmlnc/[email protected] has no problem.

[aimen08]

on wsl , with bun install .tgz not working

[ElYaiko]

Tarballs still not working :/

[medzhidov]

I think you should write in doc about ignoring binding.gyp. I have researched the whole internet to find out why deps is installed incorrectly, but I found here that it is not supported.

[Electroid]

Going to close this as we have done a lot of improvements to the reliability of bun install since this issue was created. We have lots of specific issues tracking work that still need to be done, like fixing bugs.