"Could not fetch dependencies" error when using custom registry and scoped packages

[stevenocchipinti]

Bug report

Packages affected

• [ ] sandpack-client (haven't tried without React, sorry)
• [x] sandpack-react

Description of the problem

I'm not sure if this is a sandpack bug, a verdaccio bug or I've just messed up the configuration. Apologies in the advance if this is the wrong place to post this.

What were you doing when the problem occurred?

Following the sandpack "private packages" docs to setup a custom registry (to eventually access privately hosted npm packages)

What steps can we take to reproduce the problem?

1. Configure verdaccio with the npmjs registry setup as follows (note, this was mostly in the default configuration):

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

...

packages:
  '@*/*':
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

  '**':
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

2. Run verdaccio locally with:

npx verdaccio --listen 4000

3. Configure Sandpack to use the registry as follows:

    <Sandpack
      template="react-ts"
      files={{
        "/App.tsx": code,
      }}
      customSetup={{
        dependencies: {
          "styled-components": "^5.3.6",
          // "@mui/material": "^5.10.13",
          // "@emotion/react": "^11.10.5",
          // "@emotion/styled": "^11.10.5",
        },
        npmRegistries: [
          {
            enabledScopes: [],
            limitToScopes: false,
            registryUrl: "http://localhost:4000",
          },
        ],
      }}
    />

4. This should work just fine with styled-components because it is not a scoped package Note, you should also see activity in the verdaccio proxy console

5. Remove the comments to add the MUI scoped packages

6. This will produce the following error

Could not fetch dependencies, please try again in a couple seconds: Cannot read properties of undefined (reading 'content')

Link to sandbox

Note, to use this sandbox, you'll need to follow the above steps to setup a verdaccio proxy locally on port 4000 first. In this sandbox, I've created three files:

1. NotWorking.tsx - This demonstrates the problem with a custom registry and a scoped package (@mui/material)
2. Working.tsx - This demonstrates that using a custom registry works fine with non-scoped packages (styled-components)
3. AlsoWorking.tsx - This demonstrates that scoped packages (@mui/material) still works as expected without the custom registry

Only the NotWorking example is rendered by default but these can be switched in and out in index.tsx. Hope that helps.

Your Environment

Software	Name/Version
Sandpack-client version	1.12.1
Sandpack-react version	1.17.0
Browser	Chrome
Operating System	OSX

[danilowoz]

Hey! We highly recommend only using npmRegistries configuration with the limit to scope as true. Otherwise, you won't take advantage of CDN cache and pre-optimization. Usually, the files from a registry are heavier and not optimized for production usage.

But I still would like to understand the issue. Would you like to fetch a single dependency (not scope) from your custom registry?

[stevenocchipinti]

Hello, yep, I do intend on setting limit to scope as true but I'll opt for those optimisations once I get a working proof of concept. In this case, it didn't seem to make a difference from what I could tell.

My end goal is to fetch a single scoped dependency from a private, custom registry however to demonstrate the problem I'm facing, I'm just using npmjs for this example, which results in the same error.

The problem seems to be with fetching scoped packages from a custom registry.

Non-scoped packages from a custom registry works fine. As does scoped packages from a non-custom registry.

Were you able to see the problem with the steps above?

[singerbj]

I have the exact same issue as @stevenocchipinti , @danilowoz . Here is a slightly simplified version of the issue shown on codesandbox itself: https://codesandbox.io/s/sandpack-custom-registry-forked-v79dv7?file=/src/NotWorking.tsx

Note: You need to run the example in a browser with CORS disabled to recreate it properly. This change appears to not affect the outcome of the bug itself, as I tested it in a CORS enabled browser with a proxy.

The example uses the official npmjs registry (https://registry.npmjs.org/) which you would think should work just fine. I have also tried getting a custom registry to work with my company's internal Artifactory instance as well as a locally spun up instance of Verdaccio using various uplinks. I have also tested all of the previously mentioned with and without the use of an nginx proxy - all situations recreate the issue.

One thing I noticed when comparing what appears to be the default registry for codesandbox, jsdelivr, to the other registries I tried is that the APIs return different data on some of the calls to the registries:

https://data.jsdelivr.com/v1/package/npm/react https://registry.npmjs.org/react

Ultimately I found through debugging that this line in the codesandbox-client seems to produce no content for the seemingly downloaded tar: https://github.com/codesandbox/codesandbox-client/blob/52b3b42a26e7f3e3b8345dbcf8a6982ee19279c7/packages/sandpack-core/src/npm/dynamic/fetch-protocols/utils/tar-store.ts#L27

Maybe this is an issue that should be written up in that repository? Let me know.

[singerbj]

Trying to debug this further...it looks like the issue might be with the 3rd party library isomorphic-untar-gzip. After the untar function is called on the downloaded npm package - it appears to be mangled and messed up.

That library function is used here - and only here: https://github.com/codesandbox/codesandbox-client/blob/52b3b42a26e7f3e3b8345dbcf8a6982ee19279c7/packages/sandpack-core/src/npm/dynamic/fetch-protocols/utils/tar-store.ts#L42

[Ku3mi41]

The real reason for this is the non-standard structure of the tar file. To fix the error we need to change the normalizeTar function. But my PR have no answer :|

[Ku3mi41]

Fixed by https://github.com/codesandbox/codesandbox-client/pull/8208