Create a manual test plan for releases

[aeisenberg]

This issue is about creating a simple and easy to follow manual test plan that any release manager will perform before the release.

• [ ] come up with the plan
• [ ] add it as a markdown file in this repo (either directly in the contributing guide, or as a separate file)
• [ ] update the release steps to include the manual test plan

It might be overkill, but we can write tests semi-formally in a Gherin style.

My initial thoughts are that we just create a simple set of steps that are easy to follow, but are expressed informally. We can formalize later if we have any need.

[aeisenberg]

This is meant to be a smoke test and lightly exercise all of the extension's main features.

First stab at a plan (at no point in these steps should there be an error message in a popup). These steps are for the local queries only. Someone on the Sec Exp team can fill in the portion about MRVA.

1. Install vsix file of the build you want to test.
2. Open a new workspace that has a single folder and is otherwise workspace
3. Open the CodeQL view
4. Install rails/rails database from github.com
5. CHECK: The workspace has reloaded and it is now multi-root, with the sources for rails/rails as one of the workspace roots.
6. Create a qlpack.yml file in the root of the main folder with these contents:

   name: dsp-testing/vscode-manual-testing
   version: 0.0.1
   dependencies:
     codeql/ruby-all: "*"
   

7. Create a query.ql file with the following contents:

   /**
    * @name 6 block
    * @kind problem
    * @problem.severity warning
    * @id ruby/example/six-block
    */
   
   import ruby
   
   from Block b
   where b.getNumberOfStatements() > 5 and b.getNumberOfStatements() <= 6
   select b, "This is a 6 block."
   

8. NOTICE: At this point, you may or may not have any compilation problems in your log. If you do have compilation problems, it's because the codeql/ruby-all pack is not downloaded and available in your package cache. a. If you do have compilation errors: Run the command CodeQL: Install Pack Dependencies
9. CHECK: Ensure there are no compilation errors in the problems view.
10. Run query
11. CHECK: The query results view (evantually) opens with several hundred results.
12. Select the text b.getNumberOfStatements() > 5 and b.getNumberOfStatements() <= 6
13. Run comand: CodeQL: Quick Evaluation
14. CHECK: The query results view (evantually) opens with several hundred results. Should be the same set of results in previous query run.
15. Select any ruby file in the rails/rails source folder. actioncable/lib/action_cable/engine.rb is a good one
16. Right-click -> View AST
17. CHECK: The AST view (eventually) opens with the AST for that file
18. Relax and enjoy your release.

I tried to focus on only the features that are of high importance and have been problematic in the past. There's much more we can test, but I think there are diminishing returns when we start getting to more obscure features.

[charisk]

We've started a test plan doc for the MRVA side of things. It'll be good to combine forces! It's currently a private doc (for GH employees) so I won't share here (see pinned items on our team channel) but I'm keen to make this information public.