deepeval
deepeval copied to clipboard
deepeval should return non-zero exitCode on failure
the deepeval cli command currently always exits with exitCode 0 - even if the tests fail - which makes is hard to handle failed tests in automated workflows/pipelines.
For example the following workflow on GitHub ends green/successful even if the deepeval test fails.
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Install Dependencies
run: pip install -U deepeval
- name: Run DeepEval Unit Tests
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: deepeval test run test_example.py
It would be great if the deepeval command could exit with a non-0 exitCode - see https://docs.github.com/en/actions/creating-actions/setting-exit-codes-for-actions