vscode extension test run modifies package.json after 1.67.0

[hyangah]

Our presubmit tests started to fail consistently since our test setup picked up code 1.67.0 yesterday. https://github.com/golang/vscode-go/issues/2230

Our test is basically doing

• first, run extension tests written with runTests from @vscode/test-electron
• after the extension tests succeed, run a check on the consistency between our package.json file and documentation/settings info.

After running the extension tests with 1.67.0, it looks like the second step that involves reading package.json sees a differently formatted package.json. Actually, it's not only a differently formatted package.json, but it includes a new "__metadata" property (id, publisherId, ...) at the end.

The test passes if I make the extension tests to use 1.66.2.

We can work around by arranging our presubmit test order or rewriting our package.json test logic to handle this. But, we want to understand what is going on behind the scene and why this package.json modification is necessary.

Thanks!

[connor4312]

@sandy081 this looks suspicious...

https://github.com/microsoft/vscode/blob/e09e27e716073e7cd7d3a2aaf5860380b851efb9/src/vs/platform/extensionManagement/common/extensionsScannerService.ts#L229-L241

[sandy081]

This is designed and expected. It is the metadata VS Code stores per extension while installing that can be used later for other features like updating etc.,

So I would recommend your extension to handle this.

[andyleejordan]

@sandy081 this has started breaking our extension's tests without warning, and moreover, the way it writes to package.json it's modifying the whole file (looks like whitespace changes).

[andyleejordan]

@hyangah how have you gone about working around this breaking change? I'm trying to get a release out myself and am suddenly blocked 😭

[andyleejordan]

Whelp, something else went on that broke my tests, but definitely noticed this too and will need to be careful about not committing the reformatted package.json.

[sandy081]

@andschwa Sorry about the caused breakage. I am not sure what is your test doing, but I think it's not a good idea to depend on the content of package.json, because VS Code uses it to write metadata about the extension. We do not remove any existing data but add more data.

[symmb]

What's the recommended way of dealing with the __metadata property when testing an extension with --extensionDevelopmentPath set to a version-controlled source directory? Should the changes be committed or do extension authors have to take care of reverting it (which probably means parsing package.json and clearing the property).

[andyleejordan]

I'm currently reverting it. Would love to know if we should do otherwise.

[sandy081]

Reopening this to investigate if there is something changed on VS Code while running extension tests

[remcohaszing]

I wanted to re-report this issue, because it was previously misunderstood and closed. Of course I won’t now this is open, but I already typed out reproduction steps. Perhaps they help.

Steps to Reproduce:

1. Setup a git repository (this makes the issue more clear)

   mkdir vscode-bug
   cd vscode-bug
   git init
   

2. Create a minimal VSCode extension with a test. The issue becomes more apparent if an existing extension name / publisher is used.

   package.json

   {
     "name": "vscode-eslint",
     "publisher": "dbaeumer",
     "scripts": {
       "test": "node test.mjs"
     },
     "dependencies": {
       "@vscode/test-electron": "^2.0.0"
     }
   }
   

   test.mjs

   #!/usr/bin/env node
   import {runTests} from '@vscode/test-electron'
   
   await runTests({
     extensionDevelopmentPath: new URL('.', import.meta.url).pathname,
     extensionTestsPath: new URL('index.test.js', import.meta.url).pathname
   })
   

   index.test.js

   module.exports.run = () => new Promise((resolve) => {
     // The error can be reproduced more consistently if test runs take longer
     setTimeout(resolve, 10_000)
   })
   

   .gitignore

   node_modules/
   .vscode-test/
   

3. Install and commit

   npm install
   git commit -a .
   

4. Now we actually reproduce the bug. Run the extension tests

   npm test
   

Note that running tests has modified package.json. Formatting is changed to use tabs, the final newline is stripped, and publisher metadata is added to the file.

[sandy081]

To verify:

• Create some hello world extension and open it's package.json and change the name and publisher properties to an extension that already exists in Marketplace. Eg:

{
  "name": "vscode-eslint",
  "publisher": "dbaeumer"
}

• Quit Code Insiders and Start from CLI with following arguments code-insiders --extensionDevelopmentPath <path-to-above-hello-world-extension> --user-data-dir <path-to-new-user-data-dir> --extensions-dir <path-to-new-extensions-dir>
• Make sure hello world extension's package.json is not updated with __metadata property

[ArturoDent]

This looks related to https://stackoverflow.com/questions/72983445/vscode-extension-how-to-prevent-development-related-fields-automatically-added?noredirect=1#comment128910168_72983445

which didn't have anything to do with testing the extension, but only debugging it, and getting the same metadata written to an "unsuspecting" package.json.

Will this fix handle this SO situation as well? The problem occurred very rarely so I couldn't give steps to reproduce, but I was only debugging and not running tests on the extension.

[sandy081]

Will this fix handle this SO situation as well?

Yes

[ArturoDent]

@sandy081

I have learned more about my situation of getting this extra data when debugging an extension. It happens when the extension writes to the package.json. I have a couple of extensions which allow the user to make settings that will be read and converted to commands.

I files a separate issue: https://github.com/microsoft/vscode/issues/161286