Mirror mode

[akosyakov]

it is not to look yet. Maybe we won't go this way.

We would like to mirror registry but not actual content, since storage is usually very reliable. For that we will sync periodically latest extensions based on https://open-vsx.org/sitemap.xml with ovsx. We would configure upstream to https://open-vsx.org/sitemap.xml and storage service URL to https://openvsxorg.blob.core.windows.net. AzureBlobStorageService should behave no-op if ovsx calls try to upload or remove blobs.

@amvanbaren One thing which concerned me how we can identify removed extensions based on such crawling and remove them?

[amvanbaren]

@akosyakov Yes, that's going to be a bit of a challenge. We need some sort of diff functionality.

@spoenemann Is this a desirable feature for the Open VSX server?

[spoenemann]

Yes, adding functionality to ease mirroring is definitely in scope.

[akosyakov]

I think we actually figured it out by comparing order of published versions in metadata between upstream and target. If something is out of order it should be removed. I will come back when we verified it.

The real trouble which we encountered is sync of stats, like download counters, rating, reviews and so on. Not sure whether we can reuse some existing APIs for it?

[amvanbaren]

@akosyakov The /api/{namespace}/{extension} endpoint returns that data, e.g. https://open-vsx.org/api/redhat/vscode-yaml The reviewsUrl field points to an endpoint to retrieve all reviews. The downloadCount and averageRating fields contain the stats. The question is, how in sync should these stats be?

In read-only mode, should posting a review or rating be allowed?

[akosyakov]

In read-only mode, should posting a review or rating be allowed?

It should not be allowed, but they should be in synced otherwise in VS Code it looks like no one is using anything. Is there a way to put them in via API? Alternatively maybe we can do SQL update, but not sure about consistency of data with such manipulations.

[amvanbaren]

Is there a way to put them in via API?

Not really. There's an endpoint to post a review, but it uses authentication to determine the user that posted the review. We want to get reviews and keep the original user.

With these additional requirements it sounds more like a mirror mode, rather than just read-only storage.

[amvanbaren]

TODO

• [x] Sync extension versions
• [x] Only process metadata when publishing, don't upload files to cloud storage.
• [x] Sync extension download count
• [x] Disable download counting in mirrored instance
• [x] Sync extension reviews and update average rating
• [x] Disable review posting
• [x] Disable /admin endpoints
• [x] Disable /user endpoints

[akosyakov]

@amvanbaren We have a prototype of data sync here: https://github.com/gitpod-io/openvsx/blob/master/cli/src/sync.ts @jeanp413 is looking to day how to sync download count, reviews, ratings

[akosyakov]

fyl @amvanbaren @jeanp413 did a prototype of mirroring download count and average rating here: https://github.com/gitpod-io/openvsx/commit/5b355c7ab5edef9d85c06da7e79672b58a10cc97

We learned that to mirror review posting we need to mirror users as well though. Not sure whether it is feasible from legal perspective. 🤷

[amvanbaren]

@jeanp413

// TODO(ak) we cannot just publish extensions we need to respect correpsonding flags like pre-release or target // prepackaged extension does not allow to configure it, so we need to repackage?

You don't have to. The pre-release and target flags are only used by ovsx if the extension needs to be packaged. An already packaged extension has this information in its vsixmanifest. https://github.com/eclipse/openvsx/blob/3d1bfe6ca365979a094cae79210bcf85184ce62a/cli/src/publish.ts#L37-L48

@akosyakov Well, you need the user's login name.