Add upstream for VSCodeAPI endpoints

[amvanbaren]

Fixes #417

Testing Steps

Setup

• Open this branch in Gitpod.
• Wait until the publisher process has published all test extensions.
• Stop the server.
• Configure ovsx.upstream.url in dev/application.properties.
• Run the server: ./gradlew runServer.
• Publish an extension that the upstream server also has.
• Publish an extension that the upstream server doesn't have.

All Endpoints

Endpoints:

• POST /vscode/gallery/extensionquery
• GET /vscode/asset/{namespace}/{extensionName}/{version}/{assetType}/**
• GET /vscode/item?itemName={itemName}
• GET /vscode/gallery/publishers/{namespace}/vsextensions/{extension}/{version}/vspackage
• GET /vscode/unpkg/{namespaceName}/{extensionName}/{version}/**

Steps:

• Get a local item.
• Get an upstream item.
• Get a non-existent item.

/extensionquery

• Test result merging by querying for an extension that both local and upstream have. Make sure to use a page size greater than the expected result size, so that merging actually takes place.

[benoitf]

Hello, it's not related directly to this PR, but I was wondering if OpenVSX could proxy requests for the upstream request as well ?

I suppose that for example, in an enterprise, someone configures a custom OpenVSX instance with an access to openvsx.org But clients won't have access to openvsx.org

So clients should receive a list of 'local OpenVSX instance' and then all downloads/info/etc calls are proxied to the remote upstream URL

[amvanbaren]

So clients should receive a list of 'local OpenVSX instance' and then all downloads/info/etc calls are proxied to the remote upstream URL.

@benoitf You mean rewriting an upstream response to change all upstream URLs to local server URLs? It's possible, but I don't know how common this usecase is.

Can you open a new issue where this can be discussed further?

[benoitf]

@amvanbaren yes. I think it's common in enterprise/restricted environments.

But to continue discussion I've filed https://github.com/eclipse/openvsx/issues/452

[amvanbaren]

* when I try to fetch an extension in the web UI from upstream, it displays a blank page. I believe this is a React error, since in the console it outputs `TypeError: can't convert undefined to object`.

http://localhost:3000/extension/vscodevim/vim works for me with a local instance and a Gitpod instance running as upstream . In the dev tools network tab you can see it connects to the upstream API.

@filiptronicek Can you share a URL where the console outputs TypeError: can't convert undefined to object?

when I click on the Marketplace button in VS Code, I get redirected to /vscode/item?itemName=EXT_ID. This looks like it should redirect to /extension/publisher/id, but instead it throws an HTTP 500

I've fixed this error.

[filiptronicek]

@amvanbaren opening from the latest commit it seems I cannot initiate the server because the DB is seemingly not running:

SQL State  : 08001
Error Code : 0
Message    : Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:609) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:531) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:233) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1235) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:494) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:349) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.orm.jpa.EntityManagerFactoryUtils.findEntityManagerFactory(EntityManagerFactoryUtils.java:121) ~[spring-orm-5.3.1.jar:5.3.1]
        at org.springframework.orm.jpa.JpaTransactionManager.setBeanFactory(JpaTransactionManager.java:333) ~[spring-orm-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeAwareMethods(AbstractAutowireCapableBeanFactory.java:1810) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1775) ~[spring-beans-5.3.1.jar:5.3.1]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:609) ~[spring-beans-5.3.1.jar:5.3.1]
        ... 91 common frames omitted
Caused by: org.flywaydb.core.internal.exception.FlywaySqlException: 
Unable to obtain connection from database: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

Could you please see for yourself if you are experiencing this as well?

[amvanbaren]

Could you please see for yourself if you are experiencing this as well?

@filiptronicek I didn't experience this error. I tried both locally and on Gitpod.

[filiptronicek]

@filiptronicek Can you share a URL where the console outputs TypeError: can't convert undefined to object?

I get this after visiting /extension/akosyakov/gitpod-monitor (extension is only published upstream). The desired page flashes for a couple of miliseconds and then it blanks out with the error.

[amvanbaren]

I get this after visiting /extension/akosyakov/gitpod-monitor (extension is only published upstream). The desired page flashes for a couple of miliseconds and then it blanks out with the error.

The webui and server weren't backward compatible with previous server versions. How many previous versions should the server support for upstream requests?

[filiptronicek]

@amvanbaren how backwards compatible are we currently? I think that this will mostly be used with open-vsx.org, so supporting that will cover most of the use-cases IMO.

[akosyakov]

What is preventing from landing?

[amvanbaren]

@filiptronicek

@amvanbaren are there any docs we should add for this after this is shipped?

No, the wiki already has documentation about setting up an upstream instance. Upstream support for the /vscode endpoints was missing, now the feature is complete.

@akosyakov

What is preventing from landing?

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1377 is blocking the release from being deployed to production, and because of that it's better to not merge PRs until I get confirmation that the fixed release runs without exception.