eliza icon indicating copy to clipboard operation
eliza copied to clipboard

test: Initial release of smoke/integration tests + testing framework

Open jzvikart opened this issue 2 months ago • 1 comments

Relates to:

Risks

When the proposed tests are configured to run in CI/CD environment, they need one or more API keys which are considered as secrets. It is necessary to ensure that they are not leaked through workflows being run on behalf of malicious pull requests. In addition to restrictions on running CI/CD workflows we recommend setting up a quota on those keys, and rotating them periodically.

Background

What does this PR do?

  1. This PR adds two basic tests:
  • smoke tests: checks that the project can be built and run
  • "Hello Trump": checks that the project can use one of the built-in characters (Trump) to successfully query a LLM (OpenAI).

The "Hello Trump" test requires OpenAI API keys to run. If the API keys are not available, this test will be skipped.

  1. This PR also adds initial (proof of concept) version of integration testing framework. We plan to continue development of this framework in the future as will be adding more integration tests.

Why are we doing this? Any context or related work?

The goal of these tests is to establish a quality gate that every PR should pass at a minimum. We recommend that the tests are run at least:

  • on every PR before merging it
  • every time the main/stable branch is updated

As a policy, merges to main/stable branch should not be allowed if they would break the tests.

Documentation changes needed?

The included file tests/README.md describes the procedure how to set the tests up in a CI/CD environment. The API keys should be set up the repository settings under "secrets". When present, they will automatically be added to the .env file for the duration of the workflow.

Testing

Where should a reviewer start?

This PR adds a workflow file integrationTests.yaml with two jobs. The reviewer should:

  1. Configure API keys as described in the tests/README.md
  2. Re-run the workflows and check that both test are passing
  3. Check output of jobs for any errors.

Deploy Notes

  1. Configure API keys as described in the tests/README.md
  2. Re-run the workflows and check that both test are passing
  3. Check output of jobs for any errors.
  4. Adjust workflow rules to trigger workflows as desired (on pull request, etc.)

Discord username

user98634

jzvikart avatar Dec 11 '24 15:12 jzvikart