LibraryManager Providers are unreliable

Functional impact

Builds fail because required lib files don't exist in project.

Minimal repro steps

Install files from a library (i.e. [email protected], files: dist/ladda.min.js)
Ensure it works successfully
Later on, try to restore

Expected result

Files that restore successfully once should always restore successfully

Actual result

LibMan intermittently fails with error LIB002 or LIB018, or doesn't load the correct contents of the package (i.e., index.html instead of dist/**).

Sep 05 '18 17:09 spongessuck

Thanks for raising the issue. Is there a way to reliably repro this? Are you able to post the error output?

Sep 14 '18 17:09 justcla

I don't know whether this provides any assistance, but FWIW, I'm finding that the libman restore command works reliably with cdnjs, but referencing the provider unpkg (which works within the IDE), consistently fails via the restore command, the reported errors are: [LIB002]: The "[email protected]" library could not be resolved by the "unpkg" provider [LIB002]: The "[email protected]" library could not be resolved by the "unpkg" provider [LIB002]: The "[email protected]" library could not be resolved by the "unpkg" provider

I've attached my libman.json file: libman.json.txt

Nov 22 '18 22:11 Mamikel

I've ran into similar feedback messages from Libman a while ago when I tried to deploy an app that used it. I haven't looked into it at all, but at the time, if I rebuilt the solution first, then tried the deploy again it seemed to work (implying maybe one has caching issues)? I can't provide much more detail than that in my instance.

Nov 29 '18 02:11 nickalbrecht

I have a similar behavior. You add your package in via cdnjs (because unpkg almost never works, in fact I've never personally got it it to work). Some packages you select will come down, but others fail consistently. For instance, try to add the monaco-editor and it will consistently returns 404 for half of the files. I'm starting to think it's a CDN problem all together. Example:

Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/monaco-editor/0.15.6/min-maps/vs/loader.js.map"

However, if I navigate to that URL manually and it worked. But then I did it again and it 404'd. I reloaded that exact 404 and it worked. I opened up the Chrome utilities and reloaded every few seconds and about 20% of the requests returned a 200 with the file, the rest 404'd with the same URL. This is the behavior seen when trying to restore packages with a fair amount of files. -Something- fails and then the whole thing fails. The more files there are, the more chance something 404's and it can be different every time which is making it impossible to restore some packages via this mechanism.

I haven't used this a ton but I've observed it over a serious of a few weeks.

Dec 23 '18 21:12 blakepell

As of recent this problem became worse Error LIB010 Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js" The 404 response from cdnjs is intermittent but it fails the build pipeline and costs man-hours in reaction to failed CI/CD build notifications and followup email notification about the build notifications { "version": "1.0", "defaultProvider": "cdnjs", "defaultDestination": "wwwroot/lib", "libraries": [ { "library": "[email protected]", "destination": "wwwroot/lib/bootstrap/" }]}

Are there any plans to implement a fallback mechanism?

Thank you.

Dec 24 '18 16:12 freemstr

I've just been downloading libraries with npm and adding filesystem references to files in node_modules.

On Mon, Dec 24, 2018, 11:04 freemstr <[email protected] wrote:

As of recent this problem became worse Error LIB010 Failed to download resource from " https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js" The 404 response from cdnjs is intermittent but it fails the build and costs man-hours in reaction to failed build notifications and followup email notification about the build notifications

Are there any plans to implement a fallback mechanism?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aspnet/LibraryManager/issues/370#issuecomment-449750058, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE0TkhVrJkc3kDmJydRdOfn6vuA-QS-ks5u8PsQgaJpZM4WbUgN .

Dec 24 '18 19:12 spongessuck

@justcla At 23:00 my Azure Pipeline build failed. It succeeded several hours later with the same source project Build consist of downloading clean source and building the solution Error LIB002: The "@types/[email protected]" library could not be resolved by the "unpkg" provider libman.json { "version": "1.0", "defaultProvider": "cdnjs", "defaultDestination": "JS/lib", "libraries": [ { "library": "@types/[email protected]", "destination": "TS/lib/jquery/", "provider": "unpkg" } ]}

Could this be related? https://github.com/aspnet/LibraryManager/issues/241

Jan 15 '19 14:01 freemstr

Are there any plans to implement a fallback mechanism?

I think this would be useful. Builds failing because they cant get to the cdn could be a big problem. What if someone was try to release a patch for a client who's having an issue... it needs to be sorted.

Feb 01 '19 11:02 srosam

Here the same problems. But also when it is in production that unpkg.com is sometimes very slow and unreachable. For builds this is a lesser issue locally, because we can retry this, but when having CI it will fail the builds. In production it could take 30 seconds to load the page.. Something is wrong with unpkg, you should expect CloudFlare to keep it stable.

Feb 07 '19 07:02 KoalaBear84

Maybe the same bug here. Sometimes the packages can be restored, sometimes they cannot be resolved by "unpkg". I've read that unpkg is way better than cdnjs because of its architecture. While unpkg makes npmjs packages accessible, cdnjs is barely manageable anymore these days. If unpkg is just as bad, there are no more options and I'm keeping a copy of the JavaScript files in my Git repository again, just like in the past. It's not that they are too big for that. At least they're there where my code is, and not gone half of the time.

Also, why does LibMan have to restore these files from online at every build? The files are there, there's no reason to fail the build because they couldn't be downloaded. That's just stupid.

Apr 02 '19 18:04 ygoe

And strangely, ASPNETCDN is not one of the listed package providers. You'd think Microsoft have an option to use its own CDN.

Jun 10 '19 19:06 gbjbaanb

@justcla I can frequently repro this when trying to restore material-design-icons, which contains a large number of files. It fails on a different file every time. It seems that restore does not retry individual files, and the entire operation fails if any file fails.

Jul 16 '19 20:07 kjkrum

The LIB002 is most likely due to GET requests going via HTTP protocol - in such cases unpkg returns status code 301. That's the behaviour we're seeing as per #491

LIB018 seems to be present in larger packages - the meta related to it becomes truncated as confirmed with Fiddler.

Are there any timeouts in play within Libman to prevent hangs during the restore process. It could also be race conditions. If unpkg takes more than 30s to return the full package payload does Libman simply kill the request and throw?

Jul 31 '19 10:07 saluton-mundo

Oh, maybe it has to fetch on every build because of #490. It doesn't know what it's doing or what it has done. So it has to check every time. And it even fails either way. Can't detect new files and can't download anything, then blows up either the build or the application at runtime. It's on my list of things to throw out the window now.

Jul 31 '19 11:07 ygoe

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.

Oct 23 '19 21:10 hempels

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.

Also running into the same error (also unpkg) and it's preventing production builds intermittently.

Oct 25 '19 13:10 Ghosts

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I am having the same issue as hempels (Visual studio Community 2019 16.3.6). It would be great if this was solved as for my project Libman seems to be a generally good option.

Nov 26 '19 12:11 Alex-Horlock

I got fed up with it. All of the packages I used unpkg for originally are now obtainable from jsDelivr, I believe it was added after unpkg. So now I'm just using jsDelivr instead. Still using libman mind you, I just no longer using unpkg as a package source.

Nov 26 '19 19:11 nickalbrecht

@nickalbrecht jsDelivr is not available in the selection. How did you do that?

Nov 29 '19 16:11 ygoe

@ygoe Just update Library Manager to the most recent version, it was added at a later date so earlier version won't have it.

Nov 29 '19 22:11 nickalbrecht

@ygoe which version of VS are you using? The JSDelivr provider wasn't included in VS until 16.0, but you can build a private version of the VSIX. See more here.

Nov 30 '19 02:11 jimmylewis

Tried with VS 2017. Not switched to VS 2019 yet. Waiting for a usable version (without several blocking bugs) with .NET Core 3 support (which should be 16.4).

Nov 30 '19 11:11 ygoe

You should also be able to use the CLI or the Build package if you don't have VS2019. I use Microsoft.Web.LibraryManager.Build so that my libraries restore as part of the build outside of VS. But we're kind of getting off topic :-P

Dec 03 '19 19:12 nickalbrecht

All my install library are failing to restore when i run restore client side library for my libma.js file. I am recieving this error " library could not be resolved by the "cdnjs" provider" example: { "library": "[email protected]", "destination": "wwwroot/lib/twitter-bootstrap/" }, { "library": "[email protected]", "destination": "wwwroot/lib/font-awesome/" }, { "library": "[email protected]", "destination": "wwwroot/lib/toastr.js/" }, jquery works. others coud not be able to restore. I am using visual studio 2019.

Dec 30 '19 17:12 okosodovictor

Correct, around 8 hours ago this started happening. It is because of problems at cdnjs / Cloudflare. Looks like it should is resolved almost an hour ago: https://status.cdnjs.com/incidents/0gdmkm4tl98d

Dec 30 '19 18:12 KoalaBear84

Issues solved. The package restore is working fine again.

Dec 31 '19 10:12 okosodovictor

I had a similar issue. The problem was with package name. Instead of:

{
      "library": "[email protected]",
      "destination": "wwwroot/lib/bootstrap/"
    },

I was using

{
      "library": "[email protected]",
      "destination": "wwwroot/lib/bootstrap/"
    },

The message was confusing. And it wasn't installing any of my packages.

The message should be something like The package "[email protected]" was not found in "cdnjs" repository rather than The "[email protected]" library could not be resolved by the "cdnjs" provider.

Jan 08 '20 22:01 fairking

@denis-pujdak-adm-it If I were to rephrase your point, we should be emphasizing that the library name couldn't be found separately from the name@version? E.g. The package "[email protected]" was not found in "cdnjs" repository. vs. The "bootstrap" library could not be resolved by the "cdnjs" provider (removing the version here to emphasize the name)

Jan 13 '20 21:01 jimmylewis

@jimmylewis Hm. Anyway. That was just my thought based on my experience with the instalation.

Jan 14 '20 09:01 fairking

I am also getting some intermittent errors when running dotnet build or dotnet watch run. most of the time it's fine but then every now and again I get this:

libman.json : error LIB002: The "[email protected]" library could not be resolved by the "unpkg" provider [/Users/solrevdev/code/projectnamehere/src/web/web.csproj]

The build failed. Fix the build errors and run again.

libman.json

{
    "version": "1.0",
    "defaultProvider": "cdnjs",
    "libraries": [
        {
            "provider": "unpkg",
            "library": "[email protected]",
            "files": [
                "dist/css/bootstrap.css",
                "dist/css/bootstrap.css.map",
                "dist/css/bootstrap.min.css",
                "dist/css/bootstrap.min.css.map",
                "dist/js/bootstrap.js",
                "dist/js/bootstrap.js.map",
                "dist/js/bootstrap.min.js",
                "dist/js/bootstrap.min.js.map"
            ],
            "destination": "wwwroot/lib/bootstrap/"
        },
        {
            "library": "[email protected]",
            "destination": "wwwroot/lib/bootstrap-colorpicker/dist"
        },
        {
            "library": "[email protected]",
            "files": ["jquery.min.js", "jquery.js", "jquery.min.map"],
            "destination": "wwwroot/lib/jquery/dist/"
        },
        {
            "library": "[email protected]",
            "files": ["jquery.validate.js", "jquery.validate.min.js"],
            "destination": "wwwroot/lib/jquery-validation/dist"
        },
        {
            "library": "[email protected]",
            "files": [
                "jquery.validate.unobtrusive.js",
                "jquery.validate.unobtrusive.min.js"
            ],
            "destination": "wwwroot/lib/jquery-validation-unobtrusive/"
        },
        {
            "library": "[email protected]",
            "files": ["lodash.js", "lodash.min.js"],
            "destination": "wwwroot/lib/lodash/"
        },
        {
            "library": "[email protected]",
            "files": ["vue.js", "vue.min.js"],
            "destination": "wwwroot/lib/vue/"
        }
    ]
}

OS:

▲ (macmini.local) ~ screenfetch -n
 solrevdev@macmini
 OS: 64bit Mac OS X 10.14.6 18G95
 Kernel: x86_64 Darwin 18.7.0
 Uptime: 5d 19h 7m
 Packages: 439
 Shell: zsh 5.8
 Resolution: 2560x1440
 DE: Aqua
 WM: Quartz Compositor
 WM Theme: Blue (Dark)
 Disk: 293G / 525G (57%)
 CPU: Intel Core i5-4278U @ 2.60GHz
 GPU: Intel Iris
 RAM: 8668MiB / 16384MiB

dotnet

▲ (macmini.local) ~ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.102
 Commit:    573d158fea

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.14
 OS Platform: Darwin
 RID:         osx.10.14-x64
 Base Path:   /usr/local/share/dotnet/sdk/3.1.102/

Host (useful for support):
  Version: 3.1.2
  Commit:  916b5cba26

.NET Core SDKs installed:
  2.1.804 [/usr/local/share/dotnet/sdk]
  3.1.102 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Feb 27 '20 11:02 solrevdev

LibraryManager LibraryManager copied to clipboard

Providers are unreliable

Functional impact

Minimal repro steps

Expected result

Actual result

LibraryManager
LibraryManager copied to clipboard