KCL support user behavior test

[amyXia1994]

Feature Request about kcl-test

Is your feature request related to a problem? Please describe:

• Currently, the KCL-test tool supports unit testing based on KCL Schema, but hasn't been promoted to the konfig CICD adequately
• The test tools is based on the pytest module, and by executing the kclvm -mpytest, the stacks in the konfig repository has only been smoke tested and that's not enough for either lib or app developers

Describe the feature you'd like:

Enhanced KCL test framework and tools to improve the code quality and stability, and help developers of the konfig repo gain more confidence after modifying their KCL code.

The community basically divides "configuration validation and testing" into several phases (Please note that the integration testing here is not about what the ci-test/golden files do currently):

• Unit testing and compliance verification: with no real resources created/destroyed, the unit testing just verifies the expected behaviors of functions, logical code blocks, the input parameters of the program, and rules about the attribute values

• Integration testing: real resources are created and destroyed. Usually in three steps:

  • Publish the configuration to runtime.
  • Check whether the resources created in runtime conform to expectations.
  • Clear test resources. Therefore, integration testing has a certain cost compared to unit testing

• End-to-end testing: not only involves the operation of the real resources but also verifies the impact of the configuration on app services. Therefore, end-to-end testing makes great demands on the environment. Generally problems can be detected through monitoring

Basic lib developers and end users are distinguished in terms of scenarios of configuration test:

• Basic lib developers: they are responsible for maintaining the basic models logic. Before releasing those libs, unit tests and integration tests need to be run. The integration test can be a three-step action of create-validate-destroy on the downstream configurations that depend on those libs. Therefore integration test costs more than unit test, and the high concurrency of the code change even exacerbate the cost. So the timing to run integration test is a trade off between time/resource cost and efficiency.

• End users: they are usually app owners and sres. Before applying the code changes to the production, unit test, integration test, and end-to-end test need to be run. There are two big difference between the base and end test:

  • there can be some application-level validations that only apply to certain app configuration, and these validations are also need to be checked before merged to the main branch. So that end users could gain more confidence after updating application-level configs.
  • End-to-end testing is required for end users. To avoid actual impact on services in the production environment, end-to-end testing should be performed in the dev/test environment.

ref: https://sre.google/workbook/configuration-specifics/

With the help of KCL language tools, the configuration verification in the local development - integration - end to end process is connected.

The verification and test of KCL code can be divided into three phases: local development, continuous integration, and end-to-end verification:

• Preview the front-end and back-end output of the configuration with the Kusion Preview tool and view it at any time as the configuration being written.

• Cover the local and CI stage with kcl-fmt, kcl-lint, and kcl-test tools to ensure code normalization and readability, improve configuration correctness and stability, and help detect configuration errors earlier.

• support kusion diff capability, combined with kusion deps analysis, export a list of affected downstream stacks when the base libs change, and perform kusion diff preview on that stacks; The tool can be embedded in both local and pipeline stage, supporting self-review and cross-review.

• (Intended) Kusion test supports integration testing. Developers can plan, create, destroy, update, or roll back resources in the dev/test environment to ensure the correctness of the configuration

• (Intended) The Kusion tool could combine monitoring capabilities to support end-to-end testing

Describe alternatives you've considered:

Provide kcl-test command-line tool, integrate shortcut entry in Kusion IDE. Integrate to internal/external CICD pipeline and Git hook.

Features about enhanced kcl-test

The KCL-test tool already supports the unit testing capability of Schema and has been used to test some util schemas. However, there is a lack of capability and currently tests can only be written on isolated schemas such as util Schema. The enhanced KCL-test feature set I promoted:

Feature	Scenario(what it solves)	milestone
basic test capacity	Trigger tests with cli under specific directory, and during the execution output the real-time logging, and produce a summary of all the test cases; support the following test coding interface, including the given-when-then pattern, some helper function to improve readability
tools and ecology engagement	tools: embed the kusion deps tools to support partial testing; support quick entrance in popular ide platform; support popular protocol and specs about test coverage and test report; embedded in github/gitlab action
skipped test & specify which case to test	uses can input matching regex to execute specific tests when triggered by cli; Users can test a single case when triggered in IDE; the CI pipeline can trigger specific test cases according the kusion deps result
coverage	after landing in the konfig repository, a test coverage threshold can be set to improve the code quality and ensure that the import models/schemas/attributes are under test.	support step by step: 1. first support test coverage on schema level and then detail to schema attribute level; 2. branch coverage and line coverage require the capacity of core KCLVM @Peefy @chai2010
test report	support test report specs such as xUnit, and engage to existing CICD tools to run quantitative analysis and visual presentation
multi-input of one test case	Improve test code reuse
mock	Two typical scenarios：1. kcl plugin enables access to external systems such as app meta center, cmdb, and those part of code should be mocked in ut. 2. the project_context plugin will check the directory of the entrance files and read the file content of project.yaml/stack.yaml, and that plugin is widely used in nearly all of the basic libs in konfig repo, and limits the flexibility to test basic libs since the files under test must be under a standard stack structure	the necessity and priority to be decided by users requirement
benchmark	The performance of the configuration code is quantifiable and can be used as input when other platforms wants to integrate	to be decided by users of both platform users and
fuzzing	can be used to test error-prone and complex logics and functions
performance	parallel execution of test cases	first support pseudo concurrent, that is to say, just concurrently call the compile API concurrently
single-step debug		@Peefy @chai2010

the cli interface design

todo

the test code interface design

From the perspective of the basic lib developers, test the render logic of the base app confs. This test is aimed to all stacks with dependencies to certain basic lib, that is, the real stacks in the konfig repo are test cases naturally. At the same time, the basic lib developers can also create sample stacks for testing.

Real use cases on basic libs:

the code to test - example 1:

schema CheckUseCustomLogVolume(data):
  useCustomLogVolume: bool = data?.resource?.pvcTemplates and sum([1 if t?.mountPath == "/home/admin/logs" else 0 for t in data?.resource?.pvcTemplates]) == 1

the test code would be:

# use_custom_log_volume.k
import testing
data = xxx
testing.assertEquals(data, CheckUseCustomLogVolume(data))

Real use cases on certain apps:

Applications also have their specific validation rules to help to guarantee that the future updates about the app's configuration meet certain constraints and compliance. These test focus on app-level constrains.

the code to test - example 1:

todo input from @haotian

kusion compile tool

Previously, users generate the golden files by running make check-xxx to ensure that the configuration is good to compile. In the future they can use kusion compile command to reserve the effect of smoke test.

kusion preview tool

the stdout.golden files will be eliminated (those low level data won't be maintained through the version control system). However users can check that data by running kusion preview. There will be two dimensions to preview according to user requirement: 1. the checklist of the calculated/merged frontend attributes value; 2. the final output of the backend data

Semantic level enhancement of kusion deps tool

The Kusion deps tool need to be enhanced at the semantic level to identify which stacks have dependencies on a given basic schema/attribute

• Before execution of kcl-test command, users need to know the scope to test, and that will be the input to specify the test cases.
• After modifying the logic in basic libs, lib developers need to know the impact on the downstream stacks: which stacks are affected, and the what changes happened to the final configuration data of those affected stacks, so that they can decide if the basic lib change could be released.

kusion diff tool

the stdout.golden file can be kept until the kusion diff tool can work in both local/CI stages:

by embeding the kusion deps tools, when basic lib changes, this command shows the diff preview result to help basic lib developers to self-review and cross-review.

Teachability, Documentation, Adoption, Migration Strategy:

Migration Strategy:

1. add test code on basic libs
2. support kcl-test in local and ci stages
3. support kusion diff preview in local and ci stages
4. remove the stdout.golden files

reference

• google sre workbook - Configuration Specifics: https://sre.google/workbook/configuration-specifics/
• terraform test design: https://www.terraform.io/language/modules/testing-experiment
• terraform testing: https://www.hashicorp.com/blog/testing-hashicorp-terraform
• best practice on Terraform code from Azure: https://docs.microsoft.com/en-us/azure/developer/terraform/best-practices-testing-overview
• terraform compliance: https://terraform-compliance.com/
• terratest: https://terratest.gruntwork.io/

[amyXia1994]

• provide test API: Test(testFunc TestFunc, inputs []TestCase, options TestOptions) -> []TestResult
• support sample test(only a representative subset of testcases will be tested) to save local developing time

[amyXia1994]

from shupeng:

• app level compliance test: end users like app owners need to track the backend output and to check if the backend data meet the constraint in app's level. They've developed some postscripts to do those validations. It's better to be included in KCL unit test. for example, may look like this:

import testing
testing.backend_result.when(lambda item {item.kind == "Service"}).then(lambda ip {ip == "some valid ip"})

• for unit tests on basic libs: treating all of the stacks as test cases can make basic lib developer feel safer before release, so it's meaningful and can be kept as it is for now. In the future, if basic lib testing causes too much time and has a significant efficiency effect, the concept of basic lib version management can be introduced to konfig repo. From the perspective of software development, involve version and releasing management to basic lib development. For example, one week or one day as a developing iteration and only trigger full stack test in each iteration.

[Peefy]

Debugging capability and test coverage require a separate debugging toolchain and KCL Runtime support, and separate issues can be tracked and linked here.

[amyXia1994]

design on end-to-end testing

konfig's CI pipeline can integrate Terratest to perform end to end testing.

Terratest has developed a bundle of go SDK with some helper functions and patterns to provide end-to-end test for popular runtime including terraform, k8s, packer, AWS, Azure, Google CloudPlatform, OpenPolicyAgent, docker.

The thought of terratest is mainly about using popular GPL like go, perform a set of "apply-validate-destroy" operation based on calling k8s/terraform/.. runtime API to check if a single application work in the real world or, multiple applications work together. And for checking if the application work as expected, it uses HTTP calls, accessing the cloud services' API, or other ways to connect to the service according to the runtime entity it bases on.