Exit code 0 even though tests fail

[MaikVermeulen]

We're seeing Ceedling return exit code 0 even though there are failing tests with ceedling test:all:

-----------------------
14 Tests 4 Failures 0 Ignored 
FAIL

Ceedling operations completed in 2.56 seconds
root@docker-desktop:/workspaces/myproject# echo $?
0

Ceedling version:

# ceedling version
🌱 Welcome to Ceedling!

  Ceedling => 1.0.1-fb1ce6c
  ----------------------
  /var/lib/gems/3.2.0/gems/ceedling-1.0.1/
  
  Build Frameworks
  ----------------------
       CMock => 2.6.0
       Unity => 2.6.1
  CException => 1.3.4

Our project.yml:

---
---

# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.

:project:
  :use_exceptions: FALSE
  :use_test_preprocessor: :all
  :use_auxiliary_dependencies: TRUE
  :build_root: build
#  :release_build: TRUE
  :test_file_prefix: test_
  :which_ceedling: gem
  :ceedling_version: 1.0.1
  :default_tasks:
    - test:all
    - gcov:all

#:test_build:
#  :use_assembly: TRUE

#:release_build:
#  :output: MyApp.out
#  :use_assembly: FALSE

:environment:

:extension:
  :executable: .out

# Add required files to source and include paths.
:paths:
  :test:
    - +:test/**
  :source:
    - +:src/**
    # - +:/opt/esp/idf/components/**
    - +:/opt/esp/idf/components/esp_system/**
    - +:/opt/esp/idf/components/esp_driver_gpio/**
    - +:/opt/esp/idf/components/esp_hw_support/**
    - +:/opt/esp/idf/components/soc/esp32c6/**
    - +:/opt/esp/idf/components/hal/**
    - +:/opt/esp/idf/components/esp_rom/**
  :include:
    - +:include/**
    # - +:/opt/esp/idf/components/**
    - +:/opt/esp/idf/components/esp_common/**
    - +:/opt/esp/idf/components/esp_driver_gpio/**
    - +:/opt/esp/idf/components/esp_hw_support/**
    - +:/opt/esp/idf/components/soc/esp32c6/**
    - +:/opt/esp/idf/components/hal/**
    - +:/opt/esp/idf/components/esp_rom/**
    - +:/opt/esp/idf/components/log/**
    - +:/opt/esp/idf/components/efuse/**
    - +:/opt/esp/idf/components/ieee802154/**
  :support:
    - +:test/custom_mocks/**
  :libraries: []

# ADD all project definitions to the common_defines array for unit testing
:defines:
  :common: 
    &common_defines
    [
      CONFIG_HARDWARE_VERSION=1
    ]
  :test:
    - *common_defines
    - CONFIG_MOCK_BOARD_ID=0
  :test_preprocess:
    - *common_defines
    - CONFIG_MOCK_BOARD_ID=0

:cmock:
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    - :ignore
    - :ignore_arg
    - :callback
    - :expect_any_args
  :treat_as:
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8
  :ignore:
    - esp_log.h
    - esp_efuse.h

# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  # :html_report_type:  detailed
  :reports:
    - HtmlDetailed
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90

#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use
:flags:
  :test:
    :compile:
      - -std=c2x
  :gcov:
    :compile:
      - -std=c2x

# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

:plugins:
  :enabled:
    - gcov
    # - xml_tests_report
    # - stdout_pretty_tests_report
    - module_generator

What's going on here?

[Letme]

Hm, I do not have module_generator plugin enabled. Could that be the difference? I have tried with 1.0.2-72c0935 version

[MaikVermeulen]

Hm, I do not have module_generator plugin enabled. Could that be the difference? I have tried with 1.0.2-72c0935 version

Thank you for the quick response! I also tried it with module_generator commented out, but unfortunately no luck (also not after clobber)

[mvandervoord]

Hi. I don't have the time to dig into this at the moment, so I'm leaving some thoughts here.

Memo to myself (or anyone who attempts to reproduce and investigate this): It's probably worth checking some values to see what is happening. File rakefile.rb, currently line 122 has an if statement which checks if the tasks that failed included a test or gcov run. Perhaps there are situations where this does not return true, and therefore error code handling is ignored? If that passes, then perhaps the contents of test_failures_handlers has an issue?