Add Yarn PnP support

[GGomez99]

Co-authored with @gun-yu as a result of their changes (https://github.com/microsoft/typescript-go/pull/1876) being merged in this PR

Motivation

This PR adds Plug'n'Play support natively to Typescript Go, following this issue: https://github.com/microsoft/typescript-go/issues/460 It has been reviewed and supported by @arcanis, the lead maintainer of Yarn, and the original author of Yarn PnP.

Datadog has a frontend monorepo using yarn with over 6k packages, and seeing how TS Strada struggles with our current scaling, we decided to invest time in adding a native Yarn PnP support for Typescript Go. This PnP implementation has been actively used in the IDE of more than 230 engineers at Datadog, and we're committed to fixing all issues reported to us.

Challenges

We did not integrate it in our CI yet as we still have several packages failing on build mode (most errors seem to be reported in the issues section of TS Corsa). Because the TS Corsa API is not available yet, we also couldn't integrate it properly with a fast lage setup unlike with the TS Strada API.

Changes

It's based on the main changes from the original yarn patch (https://github.com/microsoft/TypeScript/commit/99f3e130bbe02473680bfa040d131c1f7f08fcc0) that the community has been maintaining for years throughout Typescript Strada updates, except that we implemented the official PnP specification so it doesn't depend on third-party code.

Implemented features:

• PnP VFS that handles virtual folders and zip files seamlessly, with caching and fallback to the original vfs if pnp is not available
• Add PnP API and manifest handling, following the yarn PnP specification
• Initialize the PnP API every time a Host is initialized for both build and LSP modes
• Add PnP support when resolving modules in internal/module/resolver.go
• Add PnP support for auto-imports and completion at internal/modulespecifiers/specifiers.go
• Add PnP support for root types at internal/core/compileroptions.go
• Handle zip paths when going to implementation with the LSP
• Update the baseline testing framework to handle PnP when needed

Missing features:

• PnP manifest auto-refresh by watching .pnp.cjs changes

Tests

• [x] Basic PnP setup
• [x] Types from transitive dependencies
• [x] Root types loading from PnP dependencies
• [x] Completion and autoimports

[GGomez99]

@microsoft-github-policy-service agree company="Datadog"

[GGomez99]

I see my tests are not passing in CI, looking into it 🙇

[adrian-gierakowski]

is this kept up to date with master? I would like to try it out on a monorepo at work. Thanks!

[GGomez99]

is this kept up to date with master? I would like to try it out on a monorepo at work. Thanks!

@adrian-gierakowski Sure! Let me update the PR 🙇 Feel free to reach out if you find any issues!

[loynoir]

Would be nice to support go install fork, due to

• go install limitation https://github.com/golang/go/issues/50278
• related PR long time not merged https://github.com/microsoft/TypeScript/pull/35206

$ GOBIN=/tmp/tsgo-pnp go install github.com/GGomez99/typescript-go/cmd/tsgo@latest
go: github.com/GGomez99/typescript-go/cmd/tsgo@latest: version constraints conflict:
        github.com/GGomez99/[email protected]: parsing go.mod:
        module declares its path as: github.com/microsoft/typescript-go
                but was required as: github.com/GGomez99/typescript-go

$ GOBIN=/tmp/tsgo-pnp go install github.com/GGomez99/typescript-go/cmd/[email protected]/yarn-pnp
go: github.com/GGomez99/typescript-go/cmd/[email protected]/yarn-pnp: \
github.com/GGomez99/typescript-go/cmd/[email protected]/yarn-pnp: \
invalid version: version "guyllian.gomez/yarn-pnp" invalid: disallowed version string

vscode TypeScriptTeam.native-preview config

  "typescript.experimental.useTsgo": true,
  "typescript.native-preview.tsdk": "/tmp/tsgo-pnp"

[GGomez99]

Would be nice to support go install fork

@loynoir I checked the go install limitations, and it looks like I would have to maintain a separate branch where I replace all github.com/microsoft/typescript-go imports with github.com/GGomez99/typescript-go, which sounds a bit tedious (and I'm not even sure if it would work?) 😓

Would it be okay for you to build the binary manually?

git clone https://github.com/GGomez99/typescript-go.git --branch guyllian.gomez/yarn-pnp
cd typescript-go
npm run build
cp ./built/local/tsgo /tmp/tsgo-pnp

[adrian-gierakowski]

Would be nice to support go install fork

@loynoir I checked the go install limitations, and it looks like I would have to maintain a separate branch where I replace all github.com/microsoft/typescript-go imports with github.com/GGomez99/typescript-go, which sounds a bit tedious (and I'm not even sure if it would work?) 😓

Would it be okay for you to build the binary manually?

git clone https://github.com/GGomez99/typescript-go.git --branch guyllian.gomez/yarn-pnp
cd typescript-go
npm run build
cp ./built/local/tsgo /tmp/tsgo-pnp

Best to wrap this with nix to turn this into reliable one liner

It's in nixpkgs, so it should be trivial to override the arc https://github.com/NixOS/nixpkgs/blob/0fc6414b45da07a3f57a64f4ce6426b1fd015c1f/pkgs/by-name/ty/typescript-go/package.nix#L13

[loynoir]

Currently, I workaround with below

  // "typescript.experimental.useTsgo": false,
  "typescript.experimental.useTsgo": true,
  "typescript.native-preview.tsdk": ".vscode",

$ chmod +x ./.vscode/tsgo
...

#!/usr/bin/env bash
set -euo pipefail

handle() {
  local TSGO_PNP_HOME="${HOME:?}/.tsgo-pnp"
  local TSGO_PNP_GOPROXY='https://proxy.golang.org,direct'

  local TSGO_PNP_FORK='https://github.com/GGomez99/typescript-go.git'
  local TSGO_PNP_BRANCH='guyllian.gomez/yarn-pnp'
  local TSGO_PNP_MODULE='./cmd/tsgo'

  local TSGO_PNP_CLONE="${TSGO_PNP_HOME:?}/repo"
  local TSGO_PNP_GOPATH="${TSGO_PNP_HOME:?}/gopath"
  local TSGO_PNP_EXE="${TSGO_PNP_HOME:?}/bin/tsgo-pnp"

  if [ ! -e "${TSGO_PNP_EXE:?}" ]; then
    {
      hash git echo env go

      if [ ! -e "${TSGO_PNP_CLONE:?}" ]; then
        git clone \
          --depth 1 \
          "${TSGO_PNP_FORK:?}" \
          -b "${TSGO_PNP_BRANCH:?}" \
          "${TSGO_PNP_CLONE:?}"
      fi

      echo "replace github.com/microsoft/typescript-go => ${TSGO_PNP_CLONE:?}" >>"${TSGO_PNP_CLONE:?}"/go.mod

      env \
        -C "${TSGO_PNP_CLONE:?}" \
        GOPROXY="${TSGO_PNP_GOPROXY:?}" \
        GOPATH="${TSGO_PNP_GOPATH:?}" \
        go build \
        -o "${TSGO_PNP_EXE:?}" \
        -v \
        "${TSGO_PNP_MODULE:?}"

    } >&2
  fi

  exec "${TSGO_PNP_EXE:?}" "$@"
}

handle "$@"

[adrian-gierakowski]

Would be nice to support go install fork

@loynoir I checked the go install limitations, and it looks like I would have to maintain a separate branch where I replace all github.com/microsoft/typescript-go imports with github.com/GGomez99/typescript-go, which sounds a bit tedious (and I'm not even sure if it would work?) 😓 Would it be okay for you to build the binary manually?

git clone https://github.com/GGomez99/typescript-go.git --branch guyllian.gomez/yarn-pnp
cd typescript-go
npm run build
cp ./built/local/tsgo /tmp/tsgo-pnp

Best to wrap this with nix to turn this into reliable one liner

It's in nixpkgs, so it should be trivial to override the arc https://github.com/NixOS/nixpkgs/blob/0fc6414b45da07a3f57a64f4ce6426b1fd015c1f/pkgs/by-name/ty/typescript-go/package.nix#L13

done here

you can build and run it with:

nix run github:adrian-gierakowski/typescript-go-yarn-pnp -- --version

note that I only tested it on x84_64-linux