deepeval icon indicating copy to clipboard operation
deepeval copied to clipboard

deepeval should return non-zero exitCode on failure

Open olensmar opened this issue 1 month ago • 0 comments

the deepeval cli command currently always exits with exitCode 0 - even if the tests fail - which makes is hard to handle failed tests in automated workflows/pipelines.

For example the following workflow on GitHub ends green/successful even if the deepeval test fails.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install Dependencies
        run: pip install -U deepeval

      - name: Run DeepEval Unit Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_example.py 

It would be great if the deepeval command could exit with a non-0 exitCode - see https://docs.github.com/en/actions/creating-actions/setting-exit-codes-for-actions

olensmar avatar May 16 '24 07:05 olensmar