Exclude built-in extensions

[amvanbaren]

Fixes #446

[tarun-patel]

Do we know by when this will be in production?

[jeanp413]

@amvanbaren Is this ready for review? I'll test it today just in case :slightly_smiling_face:

[jeanp413]

Also, how does this affects page size? if a query with page size 50 is made, we query the db for 50 rows but then we filter the result so response could be less than 50 other than the last page :thinking:

[amvanbaren]

Also, how does this affects page size? if a query with page size 50 is made, we query the db for 50 rows but then we filter the result so response could be less than 50 other than the last page 🤔

Yes, I'll move the exclusion to the query level (database and ElasticSearch). Then the totalCount is also correct again. ElasticSearch returns totalCount as part of its search result for paging purposes.

[jeanp413]

@amvanbaren I was testing this and found this bug not related to this PR though created an issue for it https://github.com/eclipse/openvsx/issues/456 VS Code makes this query to fetch the popular extensions view but the response is 19 MB

curl 'http://localhost:8080/vscode/gallery/extensionquery' \
  -H 'accept: application/json;api-version=3.0-preview.1' \
  -H 'content-type: application/json' \
  --data-raw '{"filters":[{"criteria":[{"filterType":8,"value":"Microsoft.VisualStudio.Code"},{"filterType":12,"value":"4096"}],"pageNumber":1,"pageSize":50,"sortBy":4,"sortOrder":0}],"assetTypes":[],"flags":950}' \
  --compressed

The response is listing all the files in the node_modules folder of the extension :grimacing:

{
  "assetType": "Microsoft.VisualStudio.Code.WebResources/extension/out/client/node_modules/linebreak.js.map",
  "source": "http://localhost:8080/api/ms-python/python/2019.10.44104/file/extension/out/client/node_modules/linebreak.js.map"
},
{
  "assetType": "Microsoft.VisualStudio.Code.WebResources/extension/out/client/node_modules/node-stream-zip.js",
  "source": "http://localhost:8080/api/ms-python/python/2019.10.44104/file/extension/out/client/node_modules/node-stream-zip.js"
},
{
  "assetType": "Microsoft.VisualStudio.Code.WebResources/extension/out/client/node_modules/node-stream-zip.js.map",
  "source": "http://localhost:8080/api/ms-python/python/2019.10.44104/file/extension/out/client/node_modules/node-stream-zip.js.map"
},
{
  "assetType": "Microsoft.VisualStudio.Code.WebResources/extension/out/client/node_modules/onigasm/lib/onigasm.wasm",
  "source": "http://localhost:8080/api/ms-python/python/2019.10.44104/file/extension/out/client/node_modules/onigasm/lib/onigasm.wasm"
},
{
  "assetType": "Microsoft.VisualStudio.Code.WebResources/extension/out/client/node_modules/pdfkit.js",
  "source": "http://localhost:8080/api/ms-python/python/2019.10.44104/file/extension/out/client/node_modules/pdfkit.js"
},

[jeanp413]

Overall it works great :rocket: @amvanbaren Did you finish fixing all bugs in staging? Are all the previous changes already in production?

[amvanbaren]

@jeanp413

Did you finish fixing all bugs in staging?

Yes, the main issue was a long running db migration that caused the server pod to restart over and over due to a timeout. The timeout has been increased. I've proposed a 2 step approach where the db migrations are run as a job while the pods wait for the job to finish. We're currently looking into that.

Are all the previous changes already in production?

No, not yet. I'll check if we can move it to production on Monday, using the increased timeout.