asExternalUri doesn't work

[orshefi]

Hi Guys,

I had written a new extension which renders Graphs from matplotlib into vscode webview component, it works great on vscode desktop, but I encounter problems when trying to work with it in code-server.

I cannot state what exactly the problem is but here is the workflow:

1. my vscode extension starts and listens to requests for plot viewing on port 11111 (POST request)
2. matplotlib custom backend sends a request to the extension with the url to open in webview
3. I use vscode.env.asExternalUri tunneling in order to open the url in a webview and embed the generated address in an Iframe.
4. Usually by now the webview with the webpage would have been loaded, but instead I get and error openning localhost refuse to connect

Any ideas what went wrong?

• Web Browser: Chrome
• Local OS: Ubuntu
• Remote OS: MacOS
• Remote Architecture: x86_64
• code-server --version: v3.4.1

[code-asher]

I'm not too familiar with asExternalUri but this might mean there's something missing in our implementation. We'll need to dig further.

[orshefi]

Thanks for the response, I actually solved it from another direction.

Instead of using iframe to the rendered graph url on the host, I took its html and embedded it into the webview. I also created a translator from the WebSocket that matplotlib webagg backend creates to vscode Iframe messaging infrastructure.

It works ok, you can close this issue.

I believe soon I'll release this extension to be open source and then I'll need your help to get it into your extensions server

[code-asher]

Awesome, that sounds like a really good workaround.

I'm thinking we should leave the issue open until we fix asExternalUri since it would be ideal to have parity with local VS Code.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no activity occurs in the next 5 days.

[jsjoeio]

The Tabnine team is still facing this so we need to dig into it.

[misskniss]

If the work around is not working @code-asher @jsjoeio @Emyrk what are options for addressing this?

[jsjoeio]

I'm guessing @code-asher and I will need to add a patch to fix this then.

[code-asher]

To fully fix for the Tabnine case we will need:

1. Implement asExternalUri.
2. Use the sub-domain proxy instead of path-based proxy when available. Tabnine will only work with sub-domains since their application does not use relative paths and it does not seem they are able to fix that or provide a workaround like allowing a base URL. Users will need to set up wildcard certificates and DNS (or at the very least one for the Tabnine port).
3. Allow external proxies (like Coder's dev URLs) so it can be set up to work in environments that have their own sub-domain proxy setup and cannot delegate that role to code-server.

[ksylvan]

Any updates on this issue? I’m still running into problems using TabNine on VS Code Web.

[jsjoeio]

Chatting with their team about this. Stay tuned!

[Sominemo]

Dart & Flutter extension for VS Code uses asExternalUri too, and its DevTools iframes don't work without manual port forwarding, because it tries to connect to localhost. As far as I understand, it's related to this issue.

[jsjoeio]

Thanks for letting us know @Sominemo! That'll be helpful when we go to fix this. We may want to write our own extension that uses asExternalUri and add an e2e test to make sure it works (when we fix this).

[jsjoeio]

Notes

I have started looking into this and will share my notes here and update this comment as I go.

CONCLUSION: I successfully reproduced the bug.

Investigation: does asExternalUri work for a basic callback URL? ✅

Here are the steps I'm following to check:

1. create a new extension: npx yo code
2. open code-server (4.7.0) locally (macOS + Brave) and open extension
3. use vscode.env.asExternalUri in extension (see code below)
4. run extension (F5) or click "Run extension" in Run and Debug window
5. run command and test

      const callableUri = await vscode.env.asExternalUri(
        vscode.Uri.parse(vscode.env.uriScheme + "://test")
      );

      vscode.window.showInformationMessage(
        `your url: ${callableUri.toString()}`
      );

Expected It shows callback URL in information notification and that URL is accessible in my browser.

Actual It shows callback URL in information notification and that URL is accessible in my browser.

Investigation: does asExternalUri work for a URL like localhost:8000? ❌

Similar steps as above but with this code:

      const callableUri = await vscode.env.asExternalUri(
        vscode.Uri.parse(`http://localhost:8000)
      );

      vscode.window.showInformationMessage(
        `your url: ${callableUri.toString()}`
      );

Expected

It behaves like Codespaces and transforms the URL to be behind the a proxy such as https://jsjoeio-coder-code-server-rpvvw55xp7f4x9-8000.githubpreview.dev/. More info needed here on how this should behave.

Actual

It does not transform URL and returns http://localhost:8000.

[jsjoeio]

I've identified the Root Cause as well and have an idea for fixing this.

Root Cause

Based on my findings, the call stack looks like this:

1. extension calls asExternalUri using vscode.env.asExternalUri
2. that calls asExternalUri here
3. that calls $asExternalUri here
4. calls openerService.resolveExternalUri here

This last function -- resolveExternalUri -- tries to resolve this external uri using one the _resolvers (property on class). The problem is...there are no resolvers in the browser (at least in code-server, where i'm running with yarn watch).

"Okay...so shouldn't it throw this error?"

You would think so but if you look at the link in 3 in my call stack list, you'll notice the code isn't wrapped in a try/catch block. Therefore, that error never bubbles up.

"is that a bug?"

I would say so but I might check with @code-asher. Raised upstream: https://github.com/microsoft/vscode/issues/162770

There may be more to this root cause since I'm not sure why there wouldn't be a resolver for this external URI in the first place. I know it works in Codespaces but I can't figure out how.

Solution

I'd like to discuss with @code-asher first but I did find this. When I add resolveExternalUri here called like so:

		resolveExternalUri: async (uri: URI) => {
			const v = await Promise.resolve(URI.parse("http://localhost:8080/proxy/8000"))
			return v
		},

then my extension uses it!

If this is the correct place to patch, then next steps would be to agree upon how external uris should be resolved. One approach could be to use code-server's built-in proxy and do something like localhost:8000 -> <base-url>/proxy/8000.

[jsjoeio]

I chatted with @code-asher offline and we came up with a fix. I'm going to work on it today and hopefully push up a draft PR soon. Keep an eye out.

[jsjoeio]

Good news - PR is pretty much good to go. We should be able to wrap it up Monday assuming all goes well.

[ksylvan]

Awesome. Thank you!

[ksylvan]

Good news - PR is pretty much good to go. We should be able to wrap it up Monday assuming all goes well.

Would you like me to try testing this?

[jsjoeio]

@ksylvan yes please!

P.S. I should have time to clean up my PR today and merge 🤞🏼

[ksylvan]

What's the easiest way for me to test this?

[jsjoeio]

Good question! Easiest way that comes to mind:

1. yarn init -y
2. yarn add @coder/code-server-pr@4.7.1-5624-6ae9dec77528bc9af1b1de503d56ea5318bab99b
3. yarn code-server --auth none --port 3000

That'll load code-server with the changes. Then you could try using any basic extension that uses asExternalUri and it should work. I think Tabnine will need changes upstream after our PR is merged.

[ksylvan]

That'll load code-server with the changes. Then you could try using any basic extension that uses asExternalUri and it should work. I think Tabnine will need changes upstream after our PR is merged.

$ yarn add --verbose @coder/code-server-pr@4.7.1-5624-6ae9dec77528bc9af1b1de503d56ea5318bab99b
yarn add v1.22.19
[...]
verbose 2.436532975 Request "https://registry.yarnpkg.com/balanced-match" finished with status code 200.
verbose 2.458458046 Request "https://registry.yarnpkg.com/type-check" finished with status code 200.
verbose 2.494140741 Request "https://registry.yarnpkg.com/esutils" finished with status code 200.
[2/4] Fetching packages...
error @coder/code-server-pr@4.7.1-5624-6ae9dec77528bc9af1b1de503d56ea5318bab99b: The engine "node" is incompatible with this module. Expected version "16". Got "12.22.12"
verbose 2.83002365 Error: Found incompatible module.
    at MessageError.ExtendableBuiltin (/home/kayvan/yarn/node_modules/yarn/lib/cli.js:721:66)
    at new MessageError (/home/kayvan/yarn/node_modules/yarn/lib/cli.js:750:123)
    at checkOne (/home/kayvan/yarn/node_modules/yarn/lib/cli.js:47864:11)
    at Object.check (/home/kayvan/yarn/node_modules/yarn/lib/cli.js:47883:5)
    at /home/kayvan/yarn/node_modules/yarn/lib/cli.js:7352:73
    at Generator.next (<anonymous>)
    at step (/home/kayvan/yarn/node_modules/yarn/lib/cli.js:310:30)
    at /home/kayvan/yarn/node_modules/yarn/lib/cli.js:321:13
error Found incompatible module.

[ksylvan]

Looks like my Debian instance wasn't updated to NodeJS16. Fixed that and now seeing this:

$ yarn add @coder/code-server-pr@4.7.1-5624-6ae9dec77528bc9af1b1de503d56ea5318bab99b
yarn add v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "@coder/code-server-pr > @coder/[email protected]" has unmet peer dependency "@google-cloud/logging@^9.2.1".
[4/4] Building fresh packages...
[2/2] ⠂ @coder/code-server-pr
error /home/kayvan/code-server/node_modules/@coder/code-server-pr: Command failed.
Exit code: 1

And then when I run it, I see:

yarn run v1.22.19
$ /home/kayvan/code-server/node_modules/.bin/code-server --auth none --port 3000
[2022-10-13T19:34:54.902Z] info  code-server 4.7.1-5624-6ae9dec77528bc9af1b1de503d56ea5318bab99b 6ae9dec77528bc9af1b1de503d56ea5318bab99b
[2022-10-13T19:34:54.906Z] info  Using user-data-dir ~/.local/share/code-server
[2022-10-13T19:34:54.919Z] info  Using config file ~/.config/code-server/config.yaml
[2022-10-13T19:34:54.920Z] info  HTTP server listening on http://100.79.43.44:3000/
[2022-10-13T19:34:54.921Z] info    - Authentication is disabled
[2022-10-13T19:34:54.921Z] info    - Not serving HTTPS
Loading "minimist" failed
Error: Cannot find module 'minimist'
Require stack:
- /home/kayvan/code-server/node_modules/@coder/code-server-pr/lib/vscode/out/bootstrap-amd.js
- /home/kayvan/code-server/node_modules/@coder/code-server-pr/out/node/util.js
- /home/kayvan/code-server/node_modules/@coder/code-server-pr/out/node/cli.js
- /home/kayvan/code-server/node_modules/@coder/code-server-pr/out/node/entry.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:956:15)
    at Function.Module._load (node:internal/modules/cjs/loader:804:27)

[jsjoeio]

Hmmm...I'm not sure why it's not installing on the modules. Here is one solution: https://github.com/coder/code-server/issues/5530#issuecomment-1259443775

You could also try with npm instead. Sorry for the trouble!

[ksylvan]

I see the new code with the proxy server auto-forwarding is in code-server, do you know what needs to happen with tabnine extension to use it?

[jsjoeio]

Hmm...I'd probably have to dig in but ideally they use asExternalUri to resolve the uri for their server. Something like:

// pseudo code
const serverURL = await vscode.asExternalUri("http://localhost:3000")