cnti-testcatalog
[BUG] Update generate_config to add container_names, or deprecate and update the tests that require it
Describe the bug We should update the generate_config command to support adding & populating the container_names argument if we continue to use it. Currently, only the rolling_update, rolling_downgrade and rolling_version_change tests need this argument and they're currently excluded from the certification? So, as long as these tests remain excluded, this ticket doesn't need to be addressed.
To Reproduce Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Device (please complete the following information):
- OS [e.g. Linux, iOS, Windows, Android]
- Distro [e.g. Ubuntu]
- Version [e.g. 18.04]
- Architecture [e.g. x86, arm]
- Browser [e.g. chrome, safari]
How will this be tested? aka Acceptance Criteria (optional)
(optional: unnecessary for things like spelling errors and such)
Once this issue is address how will the fix be verified?
Additional context Add any other context about the problem here.
NOTE: you can enable higher logging level output via the command line or env var. to help with debugging
# cmd line
./cnf-testsuite -l debug test
# make sure to use -- if running from source
crystal src/cnf-testsuite.cr -- -l debug test
# env var
LOGLEVEL=DEBUG ./cnf-testsuite test
Also setting the verbose option for many tasks will add extra output to help with debugging
crystal src/cnf-testsuite.cr test_name verbose
Check usage documentation for more info about invoking commands and logging
@HashNuke @agentpoyo @denverwilliams is still issue still relevant for CNTI?
Yes issue still relevant. Tagging appropriately for contributor visibility.
Notes for contributors
The generate_config helps generate a cnf-testsuite.yml file if the cnf is a helm chart or a manifest directory. It does not currently generate any config options relevant for running the rolling_* tests as mentioned in the ticket description.
spec/setup_spec.cr has some automated tests that have example usage of this command.
We could generate these fields as comments in the resulting config file, along with example values and display a note in the command's output that these fields need to be set appropriately for the relevant tests to run (instead of being skipped).
If we go any further and try to fetch previous versions of the container images in order to set useful values for the rolling_* config fields, then we would have to handle a few more scenarios (Example: What to do if the images are from a private registry? Which containers do we use for the test?)
Even with new config generation (#2138) - this issue will still stay still relevant. As I mentioned in #1321, I think that these tests should be redesigned, and possibly, after this redesign, additional info in CNF config wouldn't be needed. On the other hand, I think, that config generation primarily serves for new testsuite users, and adding configuration for this small, non-cert test group is not that necessary.
