test icon indicating copy to clipboard operation
test copied to clipboard

Exit code of 0 after a failed test

Open cpswan opened this issue 7 months ago • 22 comments

Describe the bug in detail.

A test (or tests) fail, but the exit code for the dart test process is 0.

@gkc first noticed this on a GitHub Action run. (Full logs at logs_24055523517.zip.)

test/enrollment_test.dart: A test to verify send enroll request, deny enrollment and auth by enrollmentId should fail (failed)

image

However the job still passed.

This was initially thought to be an issue with GitHub Actions, and @gkc opened https://github.com/atsign-foundation/at_libraries/issues/586 but the behavior has since been replicated locally, so it looks like it's a dart test problem rather than a GitHub Actions problem.

Include the specific command you ran and any relevant details about the environment, including operating system, Dart SDK version, and the version of package:test you are using.

dart test --concurrency=1

Dart SDK version: 3.4.1 (stable) (Tue May 21 15:46:25 2024 +0000) on "linux_x64"

+ test 1.25.7
+ test_api 0.7.2
+ test_core 0.6.4

Whenever possible, include a full reproduction example that we can run to observe the problem.

This will need Docker Compose.

 git clone https://github.com/atsign-foundation/at_libraries.git
 cd at_libraries/
 git checkout tests-testing
 cd tests/at_onboarding_cli_functional_tests/
 echo "127.0.0.1    vip.ve.atsign.zone" | sudo tee -a /etc/hosts
 dart pub get
 sudo docker compose up -d
 dart run check_docker_readiness.dart
 dart test --concurrency=1
 echo $?

@mit-mit this is the thing I mentioned to you last week at I/O Connect, so this is the GitHub issue you asked me to open.

cpswan avatar Jul 01 '24 17:07 cpswan