[BUG] GitHub references to dependencies slow/hang

[dlockhart]

Is there an existing issue for this?

• [X] I have searched the existing issues

This issue exists in the latest npm version

• [ ] I am using the latest npm

Current Behavior

As of approximately 10AM EST today, installing using a GitHub reference on NPM 6 became slow to the point where it often simply hangs. This was not the case previously, and works on NPM 7/8.

Unfortunately there are other performance issues with the v2 lock file format also related to using GitHub references that are currently preventing an upgrade to NPM > 6.

Expected Behavior

Installing a dependency using a GitHub reference should be possible.

Steps To Reproduce

1. In NPM 6
2. Run npm install github:Brightspace/d2l-fetch or npm install Brightspace/d2l-fetch
3. Both of those either hang or take 7+ minutes to install

Interestingly, npm i git+https://github.com/Brightspace/d2l-fetch.git#semver:^2 installs in 0.7 seconds.

Environment

• npm: 6.14.17
• Node.js: 14.19.2
• OS Name: macOS 12.3.1
• System Model Name: Apple MacBook Pro (2019)
• npm config:

; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.14.17 node/v14.19.2 darwin x64"

; node bin location = /Users/<username>/.nvm/versions/node/v14.19.2/bin/node
; cwd = /Users/<username>/source
; HOME = /Users/<username>
; "npm config ls -l" to show all defaults.

[ljharb]

Seems similar to #4895, but for swapped npm versions. Maybe something on GitHub’s side?

[dlockhart]

This is happening to everyone at our company -- Mac, Windows, inside GitHub Actions runners -- everywhere. (I don't think it's a Mac-specific issue).

[marcesher]

Definitely seeing this issue, starting Friday May 13th sometime early to mid-afternoon Eastern time

Here's what I see now. Installing with git+https:

time npm install git+https://github.com/segment-integrations/analytics.js-integration-segmentio.git#35b24a7505ac05111a045ee07e75096815faaa08
...

+ @segment/[email protected]
added 56 packages from 19 contributors and audited 56 packages in 9.827s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

real	0m10.404s
user	0m1.944s
sys	0m0.701s

and with github::

Note: this isn't even installing from scratch, just running the same npm install with a different protocol

time npm install github:segment-integrations/analytics.js-integration-segmentio#35b24a7505ac05111a045ee07e75096815faaa08
...

+ @segment/[email protected]
updated 1 package and audited 86 packages in 463.228s
found 2 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

real	7m44.153s
user	0m5.764s
sys	0m1.008s

Importantly, this is with an older version... nodejs 12.22.12 / npm 6.14.16

When I try the most recent node... 16.15.0 with npm 8.5.5, I see no difference in time

[grassick]

If you run npm with -ddd you'll see the failure is attempting to run git ls-remote. e.g.

git ls-remote -h -t git://github.com/mWater/jsonql.git

This also runs slow on the command line. But replacing it with:

git ls-remote -h -t ssh://[email protected]/mWater/jsonql.git

or

git ls-remote -h -t https://github.com/mWater/jsonql.git

Are both fast.

So it's trying to use the git protocol, which is odd. It's not supported by GitHub anymore: https://github.blog/2021-09-01-improving-git-protocol-security-github/

[grassick]

Even more disturbing, if you change the dependency to explicitly use ssh:

e.g. "jsonql": "git+ssh://[email protected]:mWater/jsonql.git",

It still times out attempting to get it over git protocol!

e.g.: npm sill pacote Retrying git command: ls-remote -h -t git://github.com/mWater/jsonql.git attempt # 2

[grassick]

I have a solution that works for me:

Run git config --global url."ssh://git@".insteadOf git://

This appears to configure it to use ssh instead of the git protocol which no longer works.

[ghost]

Experiencing this issue also with Node 14.19.0 & npm 6.14.17.

Before experimenting with the workarounds, would be awesome to resolve why this started just this suddenly 🤷 ...

[Chuckytuh]

We experienced the same slowness on 6.14.11, we found out that updating npm to version 7.16.0 (at least, that was our go to version for other reasons) solved the issue., although it's not clear the root cause.

[slavoroi]

Similar issues happening to our repos with github packages.

node: 16.13.0 npm: 6.14.17 (was updated from 6.14.15) OS: Ubuntu 20.04

We started experiencing slowness since yesterday. A simple npm i takes a long long time.

*** Solved it by removing a package named: "@types/mysql2": "github:types/mysql2"

And updating mysql2 which had types.

[eldadpuzach]

node: v11.13.0 npm: v6.7.0

Our npm ci build step duration increased from ~4min to ~45min.

Solved by replacing github protocol with https at package-lock.json & package.json e.g.:

-    "angular-daterangepicker": "github:fragaria/angular-daterangepicker",
+    "angular-daterangepicker": "https://github.com/fragaria/angular-daterangepicker",

[mtrezza]

Here is what fixed the issue for me in AWS EC2 / ElasticBeanstalk deployment of a Node.js app:

Run a command in .ebextensions with:

commands:
  01-configure-git:
    command: sudo git config --system url."https://github".insteadOf "git://github"
    ignoreErrors: true # optionally, so deployment doesn't fail in case the command fails for some reason

Only checking your own package.json and changing the protocol may not be enough. If there are dependencies that are also referencing their own dependencies directly in GitHub using the git protocol. The above solution accounts for all of them.

[joeycozza]

To go along with @mtrezza 's comment. If you need to address this in Travis CI, you can add it to the before_install section of your .travis.yml file like this. That addressed the issue for us.

before_install:
  - git config --global url."https://git@".insteadOf git://

[TPXP]

The root cause of the issue is that github disabled the git:// protocol for pulling repos. See their blog article and this StackOverflow answer.

It'd be awesome to release a fix for the NPM v6 series 🙏 .

[mtrezza]

Every npm version that is still under LTS should actually receive a fix for this. It became a bug to use the git protocol and in many cases npm resolves a GitHub reference without explicit protocol to that disabled protocol.

[marcesher]

Does this mean that despite the deprecation schedule mentioned in that post, they didn't actually remove support until this past Friday May 13th?

[dlockhart]

Does this mean that despite the deprecation schedule mentioned in that post, they didn't actually remove support until this past Friday May 13th?

That's my interpretation of what happened, yeah. Maybe GitHub figured it was better to do something like that on a Friday the 13th?

[eyudin]

I patched pacote in node-v14.17.6 like this:

diff --git a/lib/node_modules/npm/node_modules/pacote/lib/fetchers/git.js b/lib/node_modules/npm/node_modules/pacote/lib/fetchers/git.js
--- a/lib/node_modules/npm/node_modules/pacote/lib/fetchers/git.js
+++ b/lib/node_modules/npm/node_modules/pacote/lib/fetchers/git.js
@@ -85,13 +85,8 @@ Fetcher.impl(fetchGit, {
 
 function hostedManifest (spec, opts) {
   return BB.resolve(null).then(() => {
-    if (!spec.hosted.git()) {
-      throw new Error(`No git url for ${spec}`)
-    }
-    return plainManifest(spec.hosted.git(), spec, opts)
-  }).catch(err => {
     if (!spec.hosted.https()) {
-      throw err
+      throw new Error(`No https url for ${spec}`)
     }
     return plainManifest(spec.hosted.https(), spec, opts)
   }).catch(err => {
@@ -99,6 +94,11 @@ function hostedManifest (spec, opts) {
       throw err
     }
     return plainManifest(spec.hosted.sshurl(), spec, opts)
+  }).catch(err => {
+    if (!spec.hosted.git()) {
+      throw err
+    }
+    return plainManifest(spec.hosted.git(), spec, opts)
   })
 }

This completely solved the problem for github shortcuts.

[david-benes]

fixed by replacing npm with yarn

[dungld-2897]

I am facing this issue and the solution is change package which need to be downloaded by npm from Github repo like this: "ckeditor4": "github:ckeditor/ckeditor4-releases#full/latest", to: "ckeditor4": "4.19.0", I don't know why this happened but this temp method help me out of this issue

[obataku]

any word on whether a fix will be backporteed to npm 6? given LTS for node 14

[jrichardsz]

Here is what fixed the issue for me in AWS EC2 / ElasticBeanstalk deployment of a Node.js app:

Run a command in .ebextensions with:

commands:
  01-configure-git:
    command: sudo git config --system url."https://github".insteadOf "git://github"
    ignoreErrors: true # optionally, so deployment doesn't fail in case the command fails for some reason

Only checking your own package.json and changing the protocol may not be enough. If there are dependencies that are also referencing their own dependencies directly in GitHub using the git protocol. The above solution accounts for all of them.

Thank men, this worked for me inside of docker:

git config --system url."https://github".insteadOf "git://github"

• Linux
• Docker
• nodejs 14
• npm 6.14.17