salt icon indicating copy to clipboard operation
salt copied to clipboard

[DOCS] Testing Packages needs further elaboration

Open dmurphy18 opened this issue 1 year ago • 1 comments

Description Using the instructions provided for 'Testing Packages' https://docs.saltproject.io/en/latest/topics/packaging/testing.html Do not contain sufficient information to allow for an external to Salt Team (VMware internal basically) utilize the instructions fully. For example: tools ts setup uses a --slug whose values relate to Golden AMI's which are not available to external users, but this is in documentation for external users.

The documentation also assumes use of tools which some options are not available to external users, and those users will go exploring the other options on tools and wonder what and how to use them.

Further the tools ts setup specifics a slug, but no key-name to access it later via tools vm ssh, so just how does one access that presumed VM that was setup. If the tool is for installing locally, then it clearly needs to be stated, and then it may prove to be destructive to the local machine depending on the tests run.

Section 'Running Tests' needs elaboration as to access the VM that was setup and clear steps on running existing package tests, or on how to write new package tests.

If no VM is setup, then why provide a slug, when platform information can be obtained from the local machine, and error issued if the local platform is not one of the supported slugs.

If errors occur during setup, need instructions or better error messaging to note the next steps, or inform the user that the setup failed, at the end of the script too, not just when the error occurred.

Suggested Fix Documentation needs updating for use by external user, without access to internal tools, and tools ts setup needs to allow for external user specified VM.

Type of documentation Salt documentation, Salt modules, the Salt Repo or the Getting Started guide.

Location or format of documentation https://docs.saltproject.io/en/latest/topics/packaging/testing.html

Additional context Road testing the instructions, while developing new package test.

dmurphy18 avatar Sep 21 '23 18:09 dmurphy18

Tasklist:

  • [ ] Allow the users to explicitly list the OS slugs available (perhaps something like tools ts list-slugs)
  • [ ] Have the tools ts setup command infer as many of the arguments as possible (for right now, this is platform and slug. We need to provide an error saying that the inferred information was incorrect if it failed at a certain spot and we did in fact infer information. For example, You need to specify one of the following as a slug ID, or perhaps point them to tools ts list-slugs in this case. The same applies for platform.
  • [ ] If errors occur in tools ts setup, we can point them to the Downloading individually section of the documentation, and explain that CICD pipeline errors may cause some of their artifacts to not exist.
  • [ ] In the Running the tests section, we should reiterate the warning about running on an isolated VM due to detructive tests, and also specify that the files need to be placed in the proper location on the machine you plan to run the tests on.
  • [ ] Add documentation on how to write new package tests or make additions to existing ones. Likely this will include explanations of important fixtures, as they do most of the heavy lifting. We welcome new package tests :)
  • [ ] Add documentation on how to filter out and run only specific package tests, and document the options available to the test-pkgs-onedir test session.
  • [ ] Address the test chunk needing to be precisely the first positional argument after --, we should probably pass it as --chunk=CHUNK.

MKLeb avatar Sep 21 '23 18:09 MKLeb