yarn.lock with keys longer than 1024 characters throw parse errors

[elado]

What version of Turborepo are you using?

1.2.0-canary.1

What package manager are you using / does the bug impact?

Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Mac

Describe the Bug

Any turbo command fails with

 ERROR  yarn.lock: could not unmarshal lockfile: yaml: line 44702: could not find expected ':'

This line in yarn.lock points to (first line in the code block):

"resolve@patch:resolve@^1.0.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.10#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.4#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.5#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.6#~builtin<compat/resolve>, resolve@patch:resolve@^1.10.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.12.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.13.1#~builtin<compat/resolve>, resolve@patch:resolve@^1.14.2#~builtin<compat/resolve>, resolve@patch:resolve@^1.15.1#~builtin<compat/resolve>, resolve@patch:resolve@^1.17.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.19.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.2.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.20.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.3.2#~builtin<compat/resolve>, resolve@patch:resolve@^1.3.3#~builtin<compat/resolve>, resolve@patch:resolve@^1.4.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.8.1#~builtin<compat/resolve>, resolve@patch:resolve@^1.9.0#~builtin<compat/resolve>":
  version: 1.21.0
  resolution: "resolve@patch:resolve@npm%3A1.21.0#~builtin<compat/resolve>::version=1.21.0&hash=07638b"
  dependencies:
    is-core-module: ^2.8.0
    path-parse: ^1.0.7
    supports-preserve-symlinks-flag: ^1.0.0
  bin:
    resolve: bin/resolve
  checksum: a0a4d1f7409e73190f31f901f8a619960bb3bd4ae38ba3a54c7ea7e1c87758d28a73256bb8d6a35996a903d1bf14f53883f0dcac6c571c063cb8162d813ad26e
  languageName: node
  linkType: hard

Expected Behavior

Turbo to be able to parse the file as it's a valid yarn.lock

To Reproduce

I believe just passing this string through the parser turbo uses should reproduce it.

[elado]

Seems like the parser struggles with strings over 1024 characters (including "). Trimming some characters from the end to make it 1024 chars works.

"resolve@patch:resolve@^1.0.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.10#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.4#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.5#~builtin<compat/resolve>, resolve@patch:resolve@^1.1.6#~builtin<compat/resolve>, resolve@patch:resolve@^1.10.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.12.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.13.1#~builtin<compat/resolve>, resolve@patch:resolve@^1.14.2#~builtin<compat/resolve>, resolve@patch:resolve@^1.15.1#~builtin<compat/resolve>, resolve@patch:resolve@^1.17.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.19.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.2.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.20.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.3.2#~builtin<compat/resolve>, resolve@patch:resolve@^1.3.3#~builtin<compat/resolve>, resolve@patch:resolve@^1.4.0#~builtin<compat/resolve>, resolve@patch:resolve@^1.8.1#~builtin<compat/resolve>, resolve@patch:resolve@^:

UPDATE:

It's a yarn issue. It implements its own yaml printer that isn't spec compliant.

Go YAML parser is 1.2 spec compliant: https://github.com/go-yaml/yaml/blob/7649d4548cb53a614db133b2a8ac1f31859dda8c/scannerc.go#L827-L834

Reported here: https://github.com/yarnpkg/berry/issues/4270 Fix here: https://github.com/yarnpkg/berry/pull/4275

[elado]

Closing as this is a yarn issue and hopefully fixed when https://github.com/yarnpkg/berry/pull/4275 is merged.

[NullVoxPopuli]

I ran in to this issue again (re-trying to see if this'll work in my monorepo).

switching to yarn/berry isn't an immediate option for me right now :thinking:

This issue: https://github.com/vercel/turborepo/issues/394

[nathanhammond]

Reopening this issue; Yarn 1.x is frozen and we need to support lockfiles that are not YAML spec compliant.

[nathanhammond]

Related, here is a Yarn 1 parser: https://github.com/yarnpkg/yarn/blob/master/src/lockfile/parse.js

[weyert]

This appears to be a port of that script: https://github.com/iseki0/go-yarnlock

[NullVoxPopuli]

I just ran in to this / was reminded of this again. Can't quite get to a pnpm migration quite yet, but it would get me around the issue in a high effort way (a much better way tho. I've grown very weary of all of yarn and yarn related problems) 🙃

[nathanhammond]

We've migrated this parsing to Rust with our own parser (instead of relying on a generic YAML parser) and we should no longer be beholden to this limit.