Integration test helpers

[aarnq]

[!warning] This is a public repository, ensure not to disclose:

• [x] personal data beyond what is necessary for interacting with this pull request, nor
• [x] business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• [x] kind/feature
• [ ] kind/improvement
• [ ] kind/deprecation
• [ ] kind/documentation
• [ ] kind/clean-up
• [ ] kind/bug
• [ ] kind/other

Optional: Mark one or more of the following that are applicable:

[!important] Breaking changes should be marked kind/admin-change or kind/dev-change depending on type Critical security fixes should be marked with kind/security

• [ ] kind/admin-change
• [ ] kind/dev-change
• [ ] kind/security
• [ ] kind/adr

What does this PR do / why do we need this PR?

This adds test helpers to setup local clusters in integration tests, and tests for local clusters itself and converted harbor tests as a poc.

To facilitate this the harness has been fully containerised through the use of the run-from-container.sh script, which works with both docker and podman. Tested to work with cypress open and it can open the UI. Tested to work with different UIDs on the host then those of the container (1000). GitHub Actions uses a different one, so as it works there it works with anything.

Now changes to tests does not require changes to the tests image:

• The build process has changed so that no artefacts from the repository is required during build.
• The generation process has changed so that it is not within the build process.
• The dependencies has become globals in the container image, instead of being installed by make.

The library structure for bats tests have changed and the main library file now has helpers to load in additional libraries, including external ones.

Changes to local-cluster.sh to provide local resolve for local clusters. It spins up a CoreDNS container to provide static entries and upstream DNS in a more managed way then relying on node-local-dns, this was required to ensure tests work as intended in GitHub Actions.

Information to reviewers

To test unit and integration tests:

# From tests/

# First start local cache else you will be rate limited.
../scripts/local-cluster.sh cache create
# Second start local resolve (will prompt for password to set DNS if you have resolvectl).
../scripts/local-cluster.sh resolve create integration.dev-ck8s.com

# Build container
make build-unit # for unit tests
make build-main # for other tests

# Takes quite a while
make run-unit-static
make run-integration
# To run only api (bats) tests on harbor
make run-integration-harbor,use-api
# To run only ui (cypress) tests on harbor
make run-integration-harbor,use-ui

Note, there will have to be a larger change to how the test harness is structured, yet I do not want to do that here but I'll note it down as a future improvement.

Tests will be restructured (including unit tests) to have directories under unit|regression|test|end-to-end which will split the test suite. This will allow us to use setup_suite local to those test suites to setup whatever is needed for the entire suite, such as setting up local clusters and deploying applications. Currently setting up and tearing down harbor for each file under integration/harbor takes about five minutes, so this would help considerably.

Checklist

• [x] Proper commit message prefix on all commits
• Change checks:
  • [x] The change is transparent
  • [ ] The change is disruptive
  • [x] The change requires no migration steps
  • [ ] The change requires migration steps
  • [ ] The change upgrades CRDs
• Metrics checks:
  • [ ] The metrics are still exposed and present in Grafana after the change
  • [ ] The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • [ ] The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • [ ] The logs do not show any errors after the change
• Pod Security Policy checks:
  • [ ] Any changed pod is covered by Pod Security Admission
  • [ ] Any changed pod is covered by Gatekeeper Pod Security Policies
  • [ ] The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Network Policy checks:
  • [ ] Any changed pod is covered by Network Policies
  • [ ] The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • [ ] The change does not cause any unnecessary Kubernetes audit events
  • [ ] The change requires changes to Kubernetes audit policy
• Falco checks:
  • [ ] The change does not cause any alerts to be generated by Falco
• Bug checks:
  • [ ] The bug fix is covered by regression tests
• Config checks:
  • [ ] The schema was updated

[aarnq]

The first successful run of integration tests from the containerised test harness :tada: And it broke unit tests :sweat_smile:

[aarnq]

So this now runs unit and integration tests in GitHub Actions so I feel like it is ready for review. Please do test to run this locally, you are required to have rootful docker or rootless podman v5 with the socket enabled, and kind to setup local clusters.

I will setup regression tests to run as integration tests do as well.

I have not tested the current state of the end-to-end tests, but I would like to try to setup minimal deployments in the integration tests to run most of them in a reproducible environment in this PR before merging.

[lunkan93]

Managed to run the integration tests locally:

❯ make run-integration
FORWARD_RUNTIME=true ../scripts/run-from-container.sh compliantkubernetes-apps-tests:main make -C tests internal-integration
[ck8s] forward your environment and/or runtime to container compliantkubernetes-apps-tests:main? [y/N]: y
make: Entering directory '/home/simon/devcluster/compliantkubernetes-apps/tests'
--- bats integration
local-clusters.bats
 ✓ local cluster has kubeconfigs
 ✓ local cluster has ready nodes
 ✓ local cluster has running calico system
 ✓ local cluster has installed minio
 ✓ local cluster has running minio
 ✓ local cluster has installed tigera operator
 ✓ local cluster has running tigera operator

7 tests, 0 failures

make: Leaving directory '/home/simon/devcluster/compliantkubernetes-apps/tests'

Only really had to give r/w permissions for all users on the docker socket to get it working.

sudo chmod 666 /var/run/docker.sock

Nice work :tada:

[aarnq]

Sort of a success :tada:

~~It is just skipping some tests that I have to debug, else this is completed.~~

[aarnq]

Ready for final review.

[aarnq]

@viktor-f do you have the time to review this?

[viktor-f]

@viktor-f do you have the time to review this?

Preferably not today, but I can make time for it tomorrow.

[aarnq]

Now this is ready for another round of reviews.