berry
berry copied to clipboard
yarnpkg
[Bug] fs functions that use buffer encoding fail
- [ ] I'd be willing to implement a fix
Describe the bug
Yarn fs reimplementation assumes fs functions only accept strings as input, instead they could be Buffer or URL for example
The error you get is
TypeError: p.match is not a function
at Function.resolveVirtual (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125678:21)
at VirtualFS.mapToBase (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125715:22)
at VirtualFS.readdirPromise (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125588:44)
at PosixFS.readdirPromise (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125588:24)
To Reproduce
const fs = require("fs");
require("./.pnp.js").setup();
const bufdir = Buffer.from(".");
fs.readdir(bufdir, { encoding: "buffer" }, x => console.log(x));
Reproduction
// Sherlock reproduction
Screenshots
If applicable, add screenshots to help explain your problem.
Environment if relevant (please complete the following information):
- OS: [e.g. OSX, Linux, Windows, ...]
- Node version [e.g. 8.15.0, 10.15.1, ...]
- Yarn version [e.g. 2.0.0-rc1, ...]
Additional context
#899 is a similar problem
node-dir npm package (2 million monthly downloads) files function fails because of this
I'm running into this issue when trying to build a project using rollup with node 17.6.0 and yarn berry v3.2.0. Any updates?
rant
My minimum reproducible repo would include `serverless` and `serverless-esbuild` which is sub-optimal, it can be easily dodged by maintainers so I am not providing it here. And I don't bother learning yarn internals enough to pin point the exact place where it does codegen for `.pnp.cjs`. Nobody else can do this as efficiently as current contributors.Here is a patch instead.
moar rant
Couldn't resist and finally took a look. Now that I'm in the rabbit hole, it's @yarnpkg/fslib who's responsible of the affected code segment.
I just don't know how tf an unexpected buffer input and an expected string input is simply expressed as PortablePath in the parameter.
Come on, guys.
Workaround
Apply the following patch in your .pnp.cjs,
@@ -32543,10 +32543,11 @@ class VirtualFS extends ProxiedFS {
mapToBase(p) {
+ const pathString = `${p}`;
if (pathString === ``)
return p;
- if (this.pathUtils.isAbsolute(p))
- return VirtualFS.resolveVirtual(p);
+ if (this.pathUtils.isAbsolute(pathString))
+ return VirtualFS.resolveVirtual(pathString);
const resolvedRoot = VirtualFS.resolveVirtual(this.baseFs.resolve(PortablePath.dot));
- const resolvedP = VirtualFS.resolveVirtual(this.baseFs.resolve(p));
+ const resolvedP = VirtualFS.resolveVirtual(this.baseFs.resolve(pathString));
return ppath.relative(resolvedRoot, resolvedP) || PortablePath.dot;
}
mapFromBase(p) {
I expect the file to be re-generated every time after running yarn add/remove, the exact lines to be patched should change.
This is a very temporary solution and this issue requires actual fix to @yarnpkg/fslib.
PR hints
For those who care enough to go for a PR, you may start here.
https://github.com/yarnpkg/berry/blob/554257087edb4a103633e808253323fb9a21250d/packages/yarnpkg-fslib/sources/VirtualFS.ts#L106
https://github.com/yarnpkg/berry/blob/554257087edb4a103633e808253323fb9a21250d/packages/yarnpkg-fslib/sources/path.ts#L9-L10
I just encountered this issue when doing rmSync(storagePath, { recursive: true }); where the directory in storagePath has a very extensive subdirectory structure. The same error is not thrown for a different, simpler directory structure. In both cases I provide the path argument as a string, so the error thrown was quite puzzling to me:
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:372:5)
at validateString (node:internal/validators:120:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4367:24)
at VirtualFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:39)
at PosixFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:24)
at URLFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:24)
at _rmdirSync (node:internal/fs/rimraf:260:21)
at rimrafSync (node:internal/fs/rimraf:193:7)
at node:internal/fs/rimraf:253:9 {
code: 'ERR_INVALID_ARG_TYPE'
}
come here by google:
yarn build
(node:14086) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:14088) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
Failed to compile.
The "path" argument must be of type string. Received an instance of Buffer
mac od yarn 3.2.0 node 18.0.0
in docker (linx vm) yarn build works...
@trollkotze I encountered the same issue as you. When investigating the stack trace, the buffer is introduced by Node where recursing over child items:
https://github.com/nodejs/node/blob/01408a5aa8d4ed4b486bb8e3d9607f3e342c78b4/lib/internal/fs/rimraf.js#L251
Yarn version: 4.0.0-rc.4
Stack trace:
[error] TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:377:5)
at validateString (node:internal/validators:119:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23673:24)
at VirtualFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:39)
at PosixFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:24)
at URLFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:24)
at _rmdirSync (node:internal/fs/rimraf:260:21)
at rimrafSync (node:internal/fs/rimraf:193:7)
at node:internal/fs/rimraf:253:9
Still reproducible with the latest yarn 4.0.0-rc.6 from sources and node 18.2.0:
➜ web git:(main) ✗ yarn --version
4.0.0-rc.6.git.20220525.hash-d1961ee
➜ web git:(main) ✗ node
Welcome to Node.js v18.2.0.
Type ".help" for more information.
> const fs = require('fs');
> require('./.pnp.cjs').setup();
> const bufdir = Buffer.from('.');
> fs.readdir(bufdir, { encoding: 'buffer' }, x => console.log(x));
> TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:377:5)
at validateString (node:internal/validators:119:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (.../web/.pnp.cjs:8397:24)
at VirtualFS.readdirPromise (.../web/.pnp.cjs:8279:44)
at PosixFS.readdirPromise (.../web/.pnp.cjs:8279:24)
at URLFS.readdirPromise (.../web/.pnp.cjs:8279:24)
at .../web/.pnp.cjs:9388:20
at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
code: 'ERR_INVALID_ARG_TYPE'
}
Using the workaround from @vicary, as above, in the meantime (many thanks!)
Not sure why this issue is labelled as unreproducible though...?
This issue likely causes this strange bug https://github.com/vitejs/vite/issues/8897 on node 18> in vite
Also have the same issue as @IgnusG with Nx where every other (uncached) build and serve fails with the same TypeError for path. Plain yarn nx reset also fails.
Environment:
- Nx 14.4.2
- Node 18.5.0
- Yarn 3.2.1 & 4.0.0-rc.11 (pnpMode: strict, nodeLinker: pnp)
- macOS 12.4 (arm)
My workaround is to revert to Node 16.16.0, which doesn't trigger the same errors, since editing .pnp.cjs doesn't persist. (Though @vicary's suggestion did work, thanks!)
`yarn nx reset` Log
yarn nx reset
> NX Resetting the Nx workspace cache and stopping the Nx Daemon.
This might take a few minutes.
> NX Daemon Server - Stopped
nx.js reset
Clears all the cached Nx artifacts and metadata about the workspace and shuts down the Nx Daemon.
Options:
--help Show help [boolean]
--version Show version number [boolean]
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:388:5)
at validateString (node:internal/validators:114:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at _rmdirSync (node:internal/fs/rimraf:260:21)
at rimrafSync (node:internal/fs/rimraf:193:7)
at node:internal/fs/rimraf:253:9 {
code: 'ERR_INVALID_ARG_TYPE'
}
`yarn nx build backend` Log
yarn nx build backend
✔ 3/3 dependent project tasks succeeded [0 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run backend:build --verbose=true
asset main.js 2.96 MiB [emitted] [big] (name: main) 1 related asset
asset seed.js 2.94 MiB [emitted] [big] (name: seed) 1 related asset
asset migrate.js 2.94 MiB [emitted] [big] (name: migrate) 1 related asset
asset static/.gitkeep 0 bytes [emitted] [from: apps/backend/src/static/.gitkeep] [copied]
webpack 5.73.0 compiled successfully (70ff4521f3265f7b)
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target build for project backend and 3 task(s) it depends on (14s)
With additional flags:
--verbose=true
yarn nx build backend --skip-nx-cache --verbose
✔ 3/3 dependent project tasks succeeded [0 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run backend:build --verbose=true
> NX The "path" argument must be of type string. Received an instance of Buffer
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:388:5)
at validateString (node:internal/validators:114:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at _rmdirSync (node:internal/fs/rimraf:260:21)
at rimrafSync (node:internal/fs/rimraf:193:7)
at node:internal/fs/rimraf:253:9
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Ran target build for project backend and 3 task(s) it depends on (9s)
With additional flags:
--verbose=true
✖ 1/4 failed
✔ 3/4 succeeded [0 read from cache]
`yarn nx serve backend` Log
> nx run backend:serve --verbose=true
asset main.js 2.96 MiB [emitted] [big] (name: main) 1 related asset
asset seed.js 2.94 MiB [emitted] [big] (name: seed) 1 related asset
asset migrate.js 2.94 MiB [emitted] [big] (name: migrate) 1 related asset
asset static/.gitkeep 0 bytes [emitted] [from: apps/backend/src/static/.gitkeep] [copied]
webpack 5.73.0 compiled successfully (70ff4521f3265f7b)
Debugger listening on ws://localhost:9229/89fa1453-d1b0-42d3-b2da-07078286c1d7
Debugger listening on ws://localhost:9229/89fa1453-d1b0-42d3-b2da-07078286c1d7
For help, see: https://nodejs.org/en/docs/inspector
[Nest] 40927 - 07/09/2022, 11:49:08 AM LOG [NestFactory] Starting Nest application...
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] DatabaseProvider dependencies initialized +60ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] GraphQLProvider dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] PassportModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] AppModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] DbConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] GqlConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] AppConfigModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] GraphQLSchemaBuilderModule dependencies initialized +17ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] GraphQLModule dependencies initialized +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +75ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] SessionModule dependencies initialized +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] UserModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [InstanceLoader] AuthModule dependencies initialized +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RoutesResolver] AppController {/api}: +6ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api, GET} route +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RoutesResolver] AuthController {/api} (version: Neutral): +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api/login, POST} (version: Neutral) route +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api/protected, GET} (version: Neutral) route +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api/logout, GET} (version: Neutral) route +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RoutesResolver] UserController {/api/user}: +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api/user/:id, GET} (version: 1) route +0ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [RouterExplorer] Mapped {/api/user/create, POST} (version: 1) route +1ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [GraphQLModule] Mapped {/api/graphql, POST} route +27ms
[Nest] 40927 - 07/09/2022, 11:49:09 AM LOG [NestApplication] Nest application successfully started +41ms
yarn start backend --verbose
> nx run backend:serve --verbose=true
> NX The "path" argument must be of type string. Received an instance of Buffer
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
at new NodeError (node:internal/errors:388:5)
at validateString (node:internal/validators:114:11)
at Object.isAbsolute (node:path:1157:5)
at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
at _rmdirSync (node:internal/fs/rimraf:260:21)
at rimrafSync (node:internal/fs/rimraf:193:7)
at node:internal/fs/rimraf:253:9
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Running target "backend:serve" failed
Failed tasks:
- backend:serve
Hint: run the command with --verbose for more details.
I forked version 3.2.1 with @vicary's patch that persists the change to .pnp.cjs: troncali/yarn-3.2.1-pnp-patch.
Yarn runs from a compiled release, so the patch has to be compiled into the release. You can do this by running:
yarn set version from sources --repository https://github.com/troncali/yarn-3.2.1-pnp-patch
Happy to create a PR for this, but unsure if this is the best way to fix the bug. If one of the maintainers wants to provide some feedback or direction on this issue I can work on something more permanent.
I'm still having this issue on the latest yarn 3.2.4 on nodejs 19.0.0 using PnP. @vicary's change fixed it but it does require editing .pnp.cjs.
I'm running fs.rmSync using -rf on a directory and it fails if that directory has another directory in it.
I'm still having this issue on the latest yarn 3.2.4
This fix isn't in v3.2.4, it hasn't been released in a stable version yet. We'll make a new stable release with the fix soon but until then you can use a canary build
yarn set version canary
or build it from source
yarn set version --branch "merceyz/release/3.3.0"
Perfect, thanks @merceyz! ❤️ Works well on canary
I was looking for release information on the PR but couldn't find any. It might be a nice addition to have something automated to comment on merged PRs in which versions they are available (at least I've seen something like this in some projects). Just an idea