ginkgo icon indicating copy to clipboard operation
ginkgo copied to clipboard

Doesn't show failed test case when -p option is set.

Open bitoku opened this issue 2 years ago • 2 comments

❯ ginkgo version                                             
Ginkgo Version 2.13.1

When these conditions are met, failed test case are not shown.

  • There are ginkgo test files and builtin unittest files.
  • some of builtin unittest files fail.
  • -p option is set

You will be able to reproduce this issue in my test repository.

https://github.com/bitoku/ginkgo-test

❯ ginkgo        
--- FAIL: TestAdd (0.00s)
    main_test.go:7: Expected 1 + 1 to equal 2
Running Suite: Ginkgo Suite - /Users/atokubi/workspace/ginkgo
=============================================================
Random Seed: 1700457476

Will run 1 of 1 specs
•

Ran 1 of 1 Specs in 0.000 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
FAIL

Ginkgo ran 1 suite in 1.540507723s

Test Suite Failed
❯ ginkgo -p
Running Suite: Ginkgo Suite - /Users/atokubi/workspace/ginkgo
=============================================================
Random Seed: 1700457506

Will run 1 of 1 specs
Running in parallel across 11 processes
•

Ran 1 of 1 Specs in 0.066 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped


Ginkgo ran 1 suite in 1.218479444s

Test Suite Failed

I know it is not supposed to have both ginkgo spec files and builtin unittest files, but it is very confusing when people want to migrate builtin unittest to ginkgo.

bitoku avatar Nov 20 '23 05:11 bitoku

It looks like Ginkgo is running the test and catching that it failed, but it's just swallowed all information about the test. Even the verbose options don't help:

$ ginkgo -p -vv --show-node-events

Running Suite: Ginkgo Suite - /tmp/tt/ginkgo-test
=================================================
Random Seed: 1700523524

Will run 1 of 1 specs
Running in parallel across 7 processes
------------------------------
• [0.000 seconds]
Add
/tmp/tt/ginkgo-test/ginkgo_test.go:8
  adds two numbers
  /tmp/tt/ginkgo-test/ginkgo_test.go:9

  Timeline >>
  > Enter [It] adds two numbers - /tmp/tt/ginkgo-test/ginkgo_test.go:9 @ 11/20/23 23:38:45.485
  < Exit [It] adds two numbers - /tmp/tt/ginkgo-test/ginkgo_test.go:9 @ 11/20/23 23:38:45.485 (0s)
  << Timeline
------------------------------

Ran 1 of 1 Specs in 0.009 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped


Ginkgo ran 1 suite in 1.177801911s

Test Suite Failed

For what it's worth, ideally this should work better. It's a reasonably common requirement to mix TestXYZ() style tests with Ginkgo tests, especially for ExampleABC() documentation tests.

blgm avatar Nov 20 '23 23:11 blgm

@blgm actually,when i run my bash script:

#!/usr/bin/env bash
set -ex

cd "$(dirname "$0")"

WORKDIR="$(pwd)"

while getopts ":c:l:" opt; do
  case $opt in
    c) CONFIG_FILE="$OPTARG"
    ;;
    l) CUSTOM_LABEL="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    exit 1
    ;;
  esac
done

if [ -z "${CONFIG_FILE}" ]; then
  echo "config file empty"
  exit 1
fi


ginkgo -v --show-node-events --trace --label-filter="${CUSTOM_LABEL}" init/... -- -c ${WORKDIR}/configs/${CONFIG_FILE}

ginkgo told me :--show-node-events provide but not defined Could you please tell me how to solve this problem

scorching-sun avatar Aug 08 '24 01:08 scorching-sun