Create groupings of tests / pre-defined test suites

[ajvb]

trafficstars

A feature that has been asked for a couple of times is the ability to have predefined groups of tests that can be run, instead of having to list each test individually within a frost test ... command.

These could be groups like "security" and "cost-optimization". As well they could be grouped by severity, especially the security related tests.

There seem to be two main reasons for wanting this:

1. When new tests are added to frost or we want to enable an existing test, a lot of copy+paste has to be done within our automation code to make sure each AWS account or GCP folder is running this test. As well, if a user who is unfamiliar with the automation wants to add a new test, there is a barrier for entry that is unnecessary.
2. For new users to Frost, there has been a desire expressed around wanting a "default set of tests" to start with. Being able to run something like frost test starter --aws-profile my-account could be a great beginner UX for Frost.

[ajvb]

@g-k @sciurus does this capture the various discussions around this well enough or did I miss stuff?

[sciurus]

Yep!

[g-k]

Sounds great! Nice summary.

[g-k]

Implementation proposal:

• [ ] add frost glob list <test path glob> which uses https://docs.python.org/3/library/glob.html to search for and print tests and is roughly equivalent to frost list | grep -i ec2
• [ ] add frost glob test <test path glob> which runs the tests and is roughly equivalent to frost test $(frost list | grep -i ec2)
• [ ] add frost pytest as an alias for frost test and deprecate frost test to use test-files or something similar as the default test command (optional)

It'd be better to use pytest markers, but that'd require moving data fetching from the pytest collect phase, which would be a larger change.

[ajvb]

@g-k Why not wrap the -m MARKEXPR pytest CLI argument? But as well, it might be worth defining the groupings we want first.