os test agent

[webern]

Create a test agent with script execution mechanism for running "Os Tests".

Here is what we have written about the concept:

OS Tests

There is a class of tests which we will call OS Tests. These tests are independent of the container orchestrator and rely on communicating with a Bottlerocket host directly. For example, a test might use SSM to call apiclient get and set commands, thereby verifying that the API is working. With OS Tests, the test code runs inside the TestSys Test Agent and interacts with the Bottlerocket host remotely. (See #419 for the other type of Bottlerocket test)

With OS Tests, the test code runs inside the TestSys Test Agent and interacts with the Bottlerocket host remotely.

The following sections describe the feature by way of describing how a Bottlerocket developer would add a new test. It starts by describing how you would do that if we did not create an os-test-agent, and then proceeds to describe how we can simplify the process by creating an os-test-agent.

Writing OS Tests

First, let's look at how a Bottlerocket OS test could be written currently using the framework. After that, let's consider ways that we can streamline the process.

OS Test Components

In order to create an OS Test right now, you would need to create these components:

• A rust binary that
• implements this interface (https://github.com/bottlerocket-os/bottlerocket-test-system/blob/3f9daa140439bb69d2850d20dc64c08187a0d4b9/agent/test-agent/src/lib.rs#L63). Basically you are implementing the run function which does the testing, then returns the test result.
• a main function that looks like this (https://github.com/bottlerocket-os/bottlerocket-test-system/blob/3f9daa140439bb69d2850d20dc64c08187a0d4b9/bottlerocket/agents/src/bin/ecs-test-agent/main.rs#L271..L287)
• The place to put your code is in this directory (https://github.com/bottlerocket-os/bottlerocket-test-system/tree/develop/bottlerocket/agents/src/bin)
• Wrap your test up in a container image
• Existing agent image definitions are in this Dockerfile (https://github.com/bottlerocket-os/bottlerocket-test-system/blob/3f9daa140439bb69d2850d20dc64c08187a0d4b9/Dockerfile#L288..L296)
• The Makefile builds these like so (https://github.com/bottlerocket-os/bottlerocket-test-system/blob/3f9daa140439bb69d2850d20dc64c08187a0d4b9/Makefile#L82..L91)

For an OS Test, you are probably going to want the instance ID. Here's how you get that with TestSys.

You will create a Configuration for your test that looks like this:

#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize, Default)]
#[serde(rename_all = "camelCase")]
pub struct OsTestConfig {
  pub instance_id: String,
}

impl Configuration for OsTestConfig {}

When creating your Test object in the cluster, you can depend-on a Resource object which represents your instance. Then you can use a templated string to get the instance ID. In YAML it looks kind of like the following, but in practice we use the testsys CLI to do all of this for us:

-----------------------------------------------
Name: my-ec2-instances
Kind: Resource
Spec:
  Agent:
    Image: ecr.aws/instance-provider:latest
    Configuration:
      ami: ami-1234567890
      -- etc
------------------------------------------------
Name: demo-test
Kind: Test
Spec:
  Agent:
    Configuration:
      instanceId: "${my-ec2-instances.instanceIds}"
      -- etc
    Image: ecr.aws/my-test-agent:lates
  Resources:
    my-ec2-instances
    -- etc

Simplified but still in Rust

We can eliminate most of the boilerplate by providing a common Bottlerocket OS testing binary. The configuration for this common entrypoint can take a list of test names to run:

struct Config {
  test_names_to_run: Vec<String>,
}


We can have a simplified interface

trait OsTest {
  fn run(/*TODO*/) -> Result<TestResult>;
}

Then you could implement that simple interface and add it to a hash map that we keep:

let mut all_our_tests = HashMap::new::<Box<dyn OsTest>>();
all_our_tests.insert("some-other-test", Box::new(SomeOtherTest::default()));
all_our_tests.insert("my-new-test", Box::new(MyNewTest::default()));
Ok(all_our_tests)

A testsys CLI invocation will send the names of the tests we want to run into the test binary's configuration:

testsys run os-tests --os-test-name some-other-test --os-test-name my-new-test

Simplified using Bash

Perhaps it's burdensome write all tests in Rust. If that's the case, there are a few ways that we can let you write your test code in bash (or any language, really).

One way would be for the above-described test agent to also be able to run bash scripts (or anything executable):

std::process::Command::new(vec!["/bin/bash", "/bin/tests/my-test-script.sh"]);

If the process exits 0, the test passed. If non-zero, it failed.

Scripts on their Own

Another possibility would be for us to create a CLI interface for communicating with testsys. Then you could write a script like this:

#!/usr/bin/env bash
testapi start

aws ssm whatever
echo something > jq [blah.whatever] = bash sucks

trap # however this is done... you could trap unexpected exits;
   testapi error "Exit code ${?} occurred - so sad"
end

if [[ something = whatver bash things ]]; then
  testapi failure "The bash thing did bad things"
else
  testapi success
end

In this scenario we would probably aggregate a directory of scripts into an image (perhaps along with the Rust-driven tests):

FROM al2
COPY ./script-tests /bin/tests

Somehow, TBD we would execute the correct script(s) for the desired test(s).