Performance regression testing

[chrisglein]

Goals and plan document (internal link)

• [ ] #9416
• [ ] #9417
  • [x] #10114 Performance pipeline sets up a Azure hosted VM with RNW secrets/variables, gets the latest published version of RNW and builds it. (3 days)
    • Copy compliance.yml
    • Rename to performance.yml
    • Remove compliance specific jobs
  • [x] #10115 Pipeline now builds, deploys, and launches the apps on the VM (2 days)
    • Build app
    • Deploy app
    • Run scenario (launch)
  • [ ] #10116 Pipeline now gathers perf traces of the app and store the results in a CSV file (6 days)
    • Run machine quieting scripts - 1 day
    • Write perf profiles - 1 day if copying existing… unknown if we actually have to write our own
      • Copy WinUI profiles
      • Copy the profile in vnext/Scripts/Tracing (Hermes specific?)
    • Extract data using perf tool + profiles - 2 days
    • Store data to CSV - 3 days
      • Figure out where CSV "lives" (temp? artifacts?) - 1 day
      • Create CSV - 1 day
      • Append data to CSV - 1 day
  • [ ] #10117 Simplifies app launch + perf gathering yml to template asking for app name (5 days)
    • Learn more yml - 1 day
    • Add run-perf-scenario.yml - 2 days
    • Rewrite HelloWorld job to use run-perf-scenario.yml - 2 days
• [ ] #9418 Perf results are now stored in a database (Unknown - 8 days?)
  • Learn MongoDB - 2 days
  • Set up database machine - 2 days
  • Create database using dashboard guidelines- 2 days
  • Automate perf data storage from CSV to database - 2 days
• [ ] #9419 Can now track day to day perf changes in graphs/dashboard (Unknown - 5 days?)
  • Figure out what metrics we want to display
  • Learn dashboard schema
• [ ] Expand test suite - test priority matrix:
  • [ ] #4872

Scenario \ Platform	RNW	WASDK (WebView2)	XAML	Electron	Flutter
Baseline	0	1	1	2	3
Real-World Example	0	1	1	2	3
Multiple Instances	1	2	2	3	4
Input	1	2	2	3	4
Collections	1	2	2	3	4
File System	2	3	3	4	5
Media	2	3	3	4	5

[chrisglein]

Notes from planning poker:

1. need to build some number of test apps (growing this should be costed separately). Start with existing apps (hello world, gallery, anything else is bonus). More apps = more comparison, but that should come after we have a pipeline that works with the most basic single app.
2. need to integrate into whatever system the perf team has (there will be some waiting for their output and evaluating whether it meets our needs). Assume what they have is new and not initially broadly tested (growing from a pile 'o scripts to a something more robust)
3. then enable that in our CI
4. how do we store and view long-term results for comparative analysis? (does the perf team have tech/practices here?) Regression testing = the end result. We need to cost the steps along the way. A plan that involves iteration here is what we need. This will expand over time from something very modest.