cli icon indicating copy to clipboard operation
cli copied to clipboard

[BUG] npm install 'my-package' silently drops extra hard links to the same inode in node_modules/my-package

Open SirKentington opened this issue 1 year ago • 3 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

This issue exists in the latest npm version

  • [X] I am using the latest npm

Current Behavior

When running npm install on a tarball or package containing multiple hard links to the same underlying file, only one of the hard links ends up getting installed into node_modules. The rest are silently dropped.

For background I am creating a package that includes zod as a bundled dependency. The system this package is built on hard links some of the zod files as they have identical contents. When I then go to install the generated tarball - either from the tgz file or from an npm repo - only the first hard link is present in node_modules. Any additional links to the same inode get silently dropped, which causes errors at runtime because files are missing.

I am aware that "don't make hard links" is a workaround. But hard links are extremely common in POSIX file systems (every named regular file in a file system is a hard link) and if npm pack and npm publish allow them to be included in a package, then npm install should treat them correctly.

Expected Behavior

I expect npm to deal with hard links in a tarball by either creating the hard links on the file system that mirror the hard links in the tarball, or by creating individual copies for each of the hard links. If it can't do that I would expect it to error. I would also expect that npm pack and npm publish would not be able to create something that could not be correctly consumed by npm install.

Steps To Reproduce

The following shell script creates a tarball with hard links, installs it, and shows that the extra hard linked file was silently dropped.

mkdir proj-with-hard-links consumer
(cd proj-with-hard-links && npm init --yes > /dev/null)
(cd consumer && npm init --yes > /dev/null)

echo '*** Creating file with a hard link ***'
cd proj-with-hard-links
touch file1.js
ln file1.js file2.js
ls -li *.js

echo '*** Packing project as tarball ***'
TARBALL=$(npm pack --pack-destination ../)
cd ../

echo '*** Installing proj as tarball in consumer ***'
cd consumer
npm install ../$TARBALL
echo '*** Only one file is present in node_modules instead of both file1.js and file2.js ***'
ls -li node_modules/proj-with-hard-links/*.js

Environment

  • npm: 10.8.1
  • Node.js: 20.8.0
  • OS Name: Mac OS 12.7.5
  • System Model Name: 13" Macbook Air 2016 - 2 GHz Dual-Core Intel Core i5
  • npm config:
npm config ls
; node bin location = /Users/thomson/.nvm/versions/node/v20.8.0/bin/node
; node version = v20.8.0
; npm local prefix = /private/tmp/hardlinks/consumer
; npm version = 10.8.1
; cwd = /private/tmp/hardlinks/consumer
; HOME = /Users/thomson
; Run `npm config ls -l` to show all defaults.

SirKentington avatar Jun 22 '24 20:06 SirKentington

mkdir proj-with-hard-links consumer (cd proj-with-hard-links && npm init --yes) (cd consumer && npm init --yes)

echo "*** Creating file with a hard link ***" cd proj-with-hard-links touch file1.js ln file1.js file2.js ls -li *.js

echo "*** Packing project as tarball ***" TARBALL=$(npm pack --pack-destination ../) cd ../

echo "Installing proj as tarball in consumer" cd consumer npm install ../$TARBALL echo "*** Only one file is present in node_modules instead of both file1.js and file2.js **" ls -li node_modules/proj-with-hard-links/.js

kellym202445 avatar Jun 23 '24 09:06 kellym202445

Hard links are tied to the inode structure of the file sysyem. when creating a talball,the underlying filesystems inode information is lost, and the tall ball format doesn't have a mechanism for preserving hard link references. Whwn extracting or installing the tallball,only one file iis written out to disk(since both hard links refer to the same inode), so the extra hard-linked file(file2.js) is silently discarded. please can you please more information or valid steps to reproduce the issue

kchindam-infy avatar Mar 26 '25 14:03 kchindam-infy

I do not believe that hard link information is lost in a standard tarball. Perhaps it is when npm creates a tarball. But when pnpm packs the tarball (or you use the tar command) the hard link info persists. If you pnpm pack and then tar -xvf the tarball you’ll see the hard linked files be preserved.

It’s when npm itself extracts the tarball to install it that it discards the hard link info, probably because it assumes that the tarball was packed by npm.

I think that npm can pack tarballs however it wants, however it should not assume that the tarball was also packed by npm, and should honour any hard link info it finds, even if it just uses the hard link info to make copies of the hard linked files.

On Wed, Mar 26, 2025 at 9:59 AM kchindam-infy @.***> wrote:

Hard links are tied to the inode structure of the file sysyem. when creating a talball,the underlying filesystems inode information is lost, and the tall ball format doesn't have a mechanism for preserving hard link references. Whwn extracting or installing the tallball,only one file iis written out to disk(since both hard links refer to the same inode), so the extra hard-linked file(file2.js) is silently discarded. please can you please more information or valid steps to reproduce the issue

— Reply to this email directly, view it on GitHub https://github.com/npm/cli/issues/7608#issuecomment-2754734962, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX22J4W7ERHBUTINJ4ERX32WK6EVAVCNFSM6AAAAABZ2RVSNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJUG4ZTIOJWGI . You are receiving this because you authored the thread.Message ID: @.***> [image: kchindam-infy]kchindam-infy left a comment (npm/cli#7608) https://github.com/npm/cli/issues/7608#issuecomment-2754734962

Hard links are tied to the inode structure of the file sysyem. when creating a talball,the underlying filesystems inode information is lost, and the tall ball format doesn't have a mechanism for preserving hard link references. Whwn extracting or installing the tallball,only one file iis written out to disk(since both hard links refer to the same inode), so the extra hard-linked file(file2.js) is silently discarded. please can you please more information or valid steps to reproduce the issue

— Reply to this email directly, view it on GitHub https://github.com/npm/cli/issues/7608#issuecomment-2754734962, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX22J4W7ERHBUTINJ4ERX32WK6EVAVCNFSM6AAAAABZ2RVSNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJUG4ZTIOJWGI . You are receiving this because you authored the thread.Message ID: @.***>

SirKentington avatar Mar 26 '25 15:03 SirKentington