Add automated system tests to help admins verify the installation and configuration of OnDemand

[ericfranz]

smgallo wrote:

While the documentation for the various components does explain how to use or access the component, there does not appear to be an automated test suite or manual process specified for each component to verify that the component is installed correctly (note that some components do instruct the installer to manually verify). This is a complex software and infrastructure package and it would be useful for each component to have a simple verification script that could perhaps test for the existence of a file or test the data returned from a REST endpoint. I can certainly appreciate the effort involved in creating and maintaining tests of this nature.

Internally OSC has discussed 2 wanting to add system tests for

1. Job Adapters - we would add these tests to https://github.com/OSC/ood_core and would provide the path to a cluster config, which would be used to submit some jobs, check status of them, output the status of all jobs, and display a report which would be helpful for adapter development and debugging.
2. Use automated system tests to create an "installation/environment dependency checklist" for Interactive Apps. I imagine from the dashboard, being able to submit a single batch job that would check for the existence of multiple dependencies for interactive apps, or maybe each interactive app having a "verification job" that runs, verifies it can start the interactive app, then closes with a user readable report available in the dashboard.

But we had not discussed having a set of system tests to verify different parts of OnDemand installation across the board.

smgallo what parts of the OnDemand installation process would you like to verify using automated tests?

1. installation of infrastructure
2. configuration of authentication
3. installation of core utility apps (shell, files, dashboard)
4. installation of core job apps (myjobs, activejobs)
5. other?

┆Issue is synchronized with this Asana task by Unito

[smgallo]

@ericfranz I think that a good start would be a way to test the core infrastructure and authentication pieces. It sounds like you've already given this some thought with the job adapters. We have a growing set of tests that we use during development and have been considering shipping a small subset of these with our software to help user's verify installation. Also, one thing that we've been discussing to assist with support requests is to provide a script that a user can run to collects various bits of information from the installation, query REST endpoints, query the database, etc. and then provide that information back to us when they submit a support ticket. The output of that might be used to verify the installation as well.

[ericfranz]

Adding some system tests to verify the job configuration does seem straight forward:

1. add a test to submit a short job, verify it was submitted, appears in the queue, and that it outputs an expected result value
2. add a test to get a status of all the running jobs and verify that the count is > 0 and certain values that all adapters will return are populated with non-null data

As far as testing authentication piece, I'm not sure exactly how to proceed. Since Apache can be configured different authentication mechanisms variations include:

1. Basic Auth
2. Federated authentication using Shibboleth or Open ID Connect
   1. Federated authentication we have cases where the OIDC provider is a Keycloak installation configured with ldap (so a single web form is presented to the user) and
   2. Federated authentication with a discover page to choose an OIDC provider (like CILogon)
   3. Federated authentication like Shibboleth that require a two factor authentication step

I could imagine a test to verify Basic Auth + user mapping but I'm not sure about the other cases. And since the other cases are recommended over Basic Auth that would limit the usefulness of such a test. Since in OnDemand authentication is two parts: Apache Auth and user mapping, maybe there would be a way to test the user mapping configuration.

[ericfranz]

@smgallo @marpierc As far as JOSS submission goes, I wanted to let you know that OnDemand components do have varying degrees of test coverage - its just tests that are useful for developers, not admins installing OnDemand (examples: https://github.com/OSC/ood_core/tree/master/spec and https://github.com/OSC/ood-dashboard/tree/master/test).

What mentioned in this issue would be required or sufficient for JOSS publication?

[smgallo]

@ericfranz I think that the two items you mentioned above would be sufficient for this submission.

Adding some system tests to verify the job configuration does seem straight forward:

• add a test to submit a short job, verify it was submitted, appears in the queue, and that it outputs an expected result value
• add a test to get a status of all the running jobs and verify that the count is > 0 and certain values that all adapters will return are populated with non-null data

[ericfranz]

We pulled in this PR to master: https://github.com/OSC/ood-dashboard/pull/360. This will be included in the 1.3 release next week.

This adds a set of Rake tasks that can be executed via the shell on the web node. For each cluster defined in the cluster config that has a valid jobs configuration section, a rake task will be auto-generated to test submitting and checking the status of a job and a single rake task is also created that will run all of the cluster specific rake tasks - so you can execute them all, or individually. The tasks will print out the arguments sent to execve when submitting the job that should be helpful with debugging.

In the future we would like to expand this to possibly verify the installation of dependencies for interactive apps and provide a web interface to this, similar to Doku's install.php https://www.dokuwiki.org/install.

[matt257]

reviewed, too difficult given constraints, maybe rephrase if we want something similar later