vscode-codeql icon indicating copy to clipboard operation
vscode-codeql copied to clipboard

Import testproj databases into workspace storage

Open aeisenberg opened this issue 1 year ago • 4 comments

Commit-by-commit review is recommended.

If a folder that ends with .testproj is encountered, assume it is a database created by a codeql test. When the user wants to import this database, copy it into workspace storage. The database can be re-imported, which first removes the old version before importing it again.

Ask user if they want to re-import outdated testproj dbs

Before running a query now, do the following:

  1. Check if the selected database is imported from a testproj
  2. If so, check the last modified time of the codeql-datase.yml file of the imported database with that of its origin.
  3. If the origin database has a file that is newer, assume that the database has been recreated since the last time it was imported.
  4. If newer, then ask the user if they want to re-import before running the query.

Also, this change appends the (test) label to all test databases in the database list.

To verify this change, do the following:

  1. In the development workspace directory, create a codeql test:
    echo "import javascript
    select 1" | tee  test.ql
    echo "dependencies:
        codeql/javascript-all: \"*\"
    extractor: javascript
    tests: test"
    echo hucairz | tee qlpack.yml
    
  2. Start the vscode dev workbench
  3. In the vscode editor, go to the test tab (looks like a beaker)
  4. Run the test. It should fail
  5. Right click on the test.testproj directory and select "(Re-)Import test database"
  6. Notice that the database is imported
  7. Run the query if you like and you should see 1 as the result.
  8. Change test to select 2
  9. Re-run the test
  10. Re-run the query
  11. This time, you should get a popup asking if you want to re-import
  12. Select yes
  13. See the results are now 2.

Checklist

  • [ ] CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
  • [ ] Issues have been created for any UI or other user-facing changes made by this pull request.
  • [ ] [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

aeisenberg avatar Mar 01 '24 09:03 aeisenberg

I'm not sure what's happening with the EPERM errors when renaming the directory on windows. It must be that the OS is hanging on to an open file handle somewhere, somehow, but I don't know where this would be happening. I could try to add a 1 second wait before renaming in case there's some sort of cleanup that needs to finish, but that feels very unsatisfactory.

aeisenberg avatar Mar 01 '24 23:03 aeisenberg

Two possibilities come to mind:

  • If you've just created the copy, then Defender might be scanning the newly-created files.
  • Does an existing CodeQL process have a lock on the DB?

adityasharad avatar Mar 01 '24 23:03 adityasharad

The next thing to try would be to add a few second wait on windows.

aeisenberg avatar Mar 03 '24 00:03 aeisenberg

OK...that seems to have worked, but it's not a great solution.

aeisenberg avatar Mar 03 '24 01:03 aeisenberg

Thanks for such a detailed review, Koen!

aeisenberg avatar Mar 13 '24 19:03 aeisenberg