test(examples): e2e test for examples

[akihikokuroda]

Which issue(s) does this pull-request address?

Closes: #62

Description

This PR changes add making examples run during e2e testing.

Checklist

• [x] I have read the contributor guide
• [x] Linting passes: yarn lint or yarn lint:fix
• [x] Formatting is applied: yarn format or yarn format:fix
• [x] Unit tests pass: yarn test:unit
• [ ] E2E tests pass: yarn test:e2e
• [ ] Tests are included
• [ ] Documentation is changed or added
• [x] Commit messages and PR title follow conventional commits

[akihikokuroda]

For now, there is only one test for "memory/agentMemory.ts" added. I want to know if this is in the right direction or not. Thanks!

[akihikokuroda]

@Tomas2D There are not changes in the /examples directory. The tests of the examples are under /tests/examples directory that executes the examples as-is and check if the examples generate any output.

[Tomas2D]

Oh. I wanted to say that I don't want 1:1 mapping between the tests/examples and the examples directories.

I would like to have one example test file, test/e2e/examples.ts, which executes (recursively) all files under the examples directory. This file will contain filters (so you can ignore some concrete examples or run them conditionally).

[akihikokuroda]

@Tomas2D Thanks for review and direction. I'll work on it.

[akihikokuroda]

yarn test:examples runs the all examples EXAMPLE_NAME=<example name pattern> yarn test:examples runs the tests that matches the pattern (I couldn't find the way to pass the pattern as an argument of the yarn command)

The interactive examples and some other are failing. I'll look into the failures.

[akihikokuroda]

In the last commit:

• move the examples.test.ts to /tests/examples
• limit the number of concurrent executions to 5

[akihikokuroda]

The last commit adds the configuration that can exclude examples from the tests in case some examples may not be expected work successfully.

[akihikokuroda]

There are still 3 example errors. I added them in the exclude list for now. They are:

Run examples/cache/fileCache.ts 964ms Run examples/llms/providers/bam_verbose.ts 1718ms Run examples/llms/providers/bam.ts 1919ms

Here is the errors.

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/cache/fileCache.ts
Error: Command failed: yarn start -- examples/cache/fileCache.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[SerializerError: Class "UnconstrainedCache" was not found!] {
  isFatal: true,
  isRetryable: false,
  context: {},
  [errors]: []
}

Node.js v20.17.0

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/llms/providers/bam_verbose.ts
Error: Command failed: yarn start -- examples/llms/providers/bam_verbose.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[LLMError: LLM has occurred an error!] {
  isFatal: false,
  isRetryable: false,
  context: {},
  [errors]: [
    HttpError: Model not found
        at clientErrorWrapper (file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/p-queue-compat/dist/esm/index.js:118:26 {
      error: 'Not Found',
      status_code: 404,
      extensions: {
        code: 'NOT_FOUND',
        state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
      },
      [cause]: {
        status_code: 404,
        error: 'Not Found',
        message: 'Model not found',
        extensions: {
          code: 'NOT_FOUND',
          state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
        }
      }
    }
  ]
}

Node.js v20.17.0

 ❯ clientErrorWrapper node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/llms/providers/bam.ts
Error: Command failed: yarn start -- examples/llms/providers/bam.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[LLMError: LLM has occurred an error!] {
  isFatal: false,
  isRetryable: false,
  context: {},
  [errors]: [
    HttpError: Model not found
        at clientErrorWrapper (file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/p-queue-compat/dist/esm/index.js:118:26 {
      error: 'Not Found',
      status_code: 404,
      extensions: {
        code: 'NOT_FOUND',
        state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
      },
      [cause]: {
        status_code: 404,
        error: 'Not Found',
        message: 'Model not found',
        extensions: {
          code: 'NOT_FOUND',
          state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
        }
      }
    }
  ]
}

Node.js v20.17.0

 ❯ clientErrorWrapper node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13

[Tomas2D]

• Regarding BAM (llama 3.1 is accessible only to some users). Let's ignore it for now. I will resolve this internally.
• Change EXAMPLE_NAME to INCLUDE or INCLUDE_PATTERN
• Add EXCLUDE or EXCLUDE_PATTERN optional env and use ignore property in the glob function
• What wraps my head around is this not working test examples/cache/fileCache.ts. It works on my machine.
  • Try to 1. run the example without vitest (yarn start examples/cache/fileCache.ts or 2. replace os.tmpdir with some different static path.

[akihikokuroda]

The environment variable names are now "INCLUDE_PATTERN" and "EXCLUDE_PATTERN". The fileCache.ts works by replacing the os.tmpdir for me. Thanks!

[Tomas2D]

The os.tmpdir() should be cross-platform. Which operating system are you using? This should work

[akihikokuroda]

It's M1 Mac. Sonoma 14.5

[akihikokuroda]

There are discussions about os.tmpdir() on Mac OS.