Ceedling Performance issue with ceedling 1.0.1

Hello,

I noticed a drop in performance , when updating from Version 1.0.0 to 1.0.1.

A test which takes 15 seconds with Version 1.0.0, now takes 42 seconds with 1.01. Switching back to Version 1.0.0 results in the old performance, which lets me assume that is in an issue with the new version.

The test is simple and so far only mocks some functions. So my guess is it could have something to do with the mock framework. I am using the fff framework. It "feels" like compiling takes a lot longer than before. But I have to check again.

Also I was not able to switch between ceedling versions in the project file.

:which_ceedling: gem
  :ceedling_version: 1.0.0

still calls ceedling-1.0.1. After removing version 1.0.1 gem calls ceedling version 1.0.0.

Jan 31 '25 11:01 JoeLowtech

Interesting! I'm not sure what would be taking longer between these two versions. The biggest core issues we addressed are changes to the way preprocessing is handled. Out of curiosity, do you have preprocessing enabled?

If you don't mind, can you run the following:

ceedling --logfile v100.txt test:all on version 1.0.0
ceedling --logfile v101.txt test:all on version 1.0.1

(I'm going to be doing this too, but it's always interesting to see this on multiple platforms)

This will give you a v100.txt and v101.txt file which have detailed logs of all the timing of all the steps. It'll be interesting to see what is taking so much time (I admit this can be a tedious process to analyze. Most often, it's small steps that happen many many times that take longer!)

For your side-issue above, I'll have to update some documentation. The ceedling_version field makes no attempt to run a particular version of ceedling. It tell you which version of ceedling generated this project.yml. At this moment, that's all it does. In the long run, the intention is to use this information to help the user. We haven't decided exactly how far to push this, since project.yml files are intentionally human editable... but we're considering automatically providing warnings / updates about feature changes when the versions don't match, etc.

If you have both installed as gems, what you CAN do is specify it as part of the call to ceedling:

ceedling _1.0.0_ version
ceedling _1.0.1_ version

running the two lines above will show you that it's calling two different versions of ceedling. :)

Jan 31 '25 14:01 mvandervoord

Hello, I did run the commands with the two versions:

I did run only one test with cmd ceedling test:pattern[minimal] . Running all tests would fail because some tests do not compile atm( missing includes ).

My findings:

I can reproduce the behavior, meaning Version 1.0.0. does complete faster than Version 1.0.1
I do have :use_test_preprocessor: :all in the project file. Switching to :use_test_preprocessor: :none decreases the exectution time meaning, the test runs faster, but it is still a lot slower than Version 1.0.0 (from 42 seconds to 27 seconds vs 16 seconds)
I noticed that compilation is the part which changes the most. It looks like in Version 1.0.0 compilation is done much faster than in version 1.0.1. Maybe not all cores are used?
A new error did sprout (of course it did, why should there only be one :D). In Version 1.0.0. rerunning the test produces this error: EXCEPTION: NameError ==> CMock >> uninitialized constant FffMockGenerator::StringIO Using clobber and running the test does work. Rerunning throws the error again. This might be completly unrelated to the performance issue. This error occured when I switched from 1.0.1 to 1.0.0, but my "old" Version 1.0.0. did not throw this error. My old version was a 1.0.0. prerelease so maybe something did change here. In Version 1.0.1 this error does not happen.

v100.txt v101.txt

Some informations to my system: I am using Win 11 with Terminal-app and ruby

Feb 03 '25 07:02 JoeLowtech

@JoeLowtech -- thanks for the thorough investigation! We really appreciate it!

To get around the redefinition problem, you can patch the gem itself so that it fixes this issue (this was one of the problems fixed in 1.0.1). Open fff.rb in the plugins folder. Near line 87-ish, you'll see the following:

# Switch out the CMock with FFF Mock Generator
require "cmock"
RealCMock = CMock
CMock = FffCMockWrapper

It wants to be this instead:

# Switch out the CMock with FFF Mock Generator
if !self.class.const_defined?(:RealCMock)
  if !self.class.const_defined?(:CMock)
    require "cmock"
    RealCMock = CMock
  end
  self.class.send(:remove_const, :CMock)
  CMock = FffCMockWrapper
end

(It's checking to see if you've already created the class for wrapping FFF with a CMock-like interface, instead of blindly creating the class each time)

Back to the main topic: I agree with your assessment of what the logs appear to be telling us. It seems like 1.0.1 is compiling each file in single-file instead of parallel on your system. This is interesting. As far as I know, there isn't any difference in the multi-tasking features of the two. Clearly I have some investigating to do here.

Feb 03 '25 20:02 mvandervoord

@mvandervoord Glad I could. If you need more information just let me know.

Feb 04 '25 10:02 JoeLowtech

Hi @JoeLowtech -- I believe I have restored the previous functionality with regard to compiling. I'm still seeing a slowdown when running the tests themselves. I'll check into this... but if you are able to verify the pre-release 1.0.2 version, I'd appreciate it.

Feb 11 '25 00:02 mvandervoord

@mvandervoord I did run the pre-release but it was no improvement. Time is 30s vs 58s The absolute timing did change because my tests evolved but I still see the same behaviour. When I have the time I will create a totally fresh project and check again. If the behaviour is different I might be able to narrow it down to something related in the project.yml.

v100.txt v102.txt

Feb 11 '25 12:02 JoeLowtech

@mvandervoord I did run a fresh example temp_sensor project. This project does build with multiple threads. I did modify some tests to run with fff as mock plugin but it did run as expected too.

If it helps I can post my project.yml

Feb 13 '25 12:02 JoeLowtech

Yes, if you're able to post your project.yml that would definitely be helpful. Perhaps it's a combination of features I haven't sufficiently validated?

Feb 13 '25 14:02 mvandervoord

@mvandervoord Here is the project.yml I use mixins for separate test configs. The mixins contain additional src/include path and defines

# =========================================================================
#   Ceedling - Test-Centered Build System for C
#   ThrowTheSwitch.org
#   Copyright (c) 2010-24 Mike Karlesky, Mark VanderVoord, & Greg Williams
#   SPDX-License-Identifier: MIT
# =========================================================================

---
:project:
  # how to use ceedling. If you're not sure, leave this as `gem` and `?`
  :which_ceedling: gem
  :ceedling_version: '?'

  # optional features. If you don't need them, keep them turned off for performance
  :use_mocks: TRUE
  :use_test_preprocessor: :all
  :use_deep_preprocessor: :none  # options are :none, :mocks, :tests, or :all
  :use_backtrace: :gdb
  :use_decorators: :auto #Decorate Ceedling's output text. Your options are :auto, :all, or :none

  # tweak the way ceedling handles automatic tasks
  :build_root: build
  :test_file_prefix: test_
  :default_tasks:
     - test:all

  # performance options. If your tools start giving mysterious errors, consider 
  # dropping this to 1 to force single-tasking
  :test_threads: 12
  :compile_threads: 12

  # enable release build (more details in release_build section below)
  :release_build: FALSE

# Specify where to find mixins and any that should be enabled automatically
:mixins:
  :enabled:
    - V8Bus_to_mj_utest_config
    - V8Bus_master_utest_config
    - conbus_master_v8bus_utest_config
    - protocol_utest_config
    - con_bus_v8bus_common_utest_config
    - device_utest_config
    - V8Bus_node_utest_config
    - CircularBUffer_no_sem_utest_config
    
  :load_paths:
    - src/V8Bus_master/test
    - src/Communication/test
    - src/ConBus/test
    - src/Device/test
    - src/V8Bus_node/test
    - src/CircularBuffer_no_sem/test

# further details to configure the way Ceedling handles test code
:test_build:
  :use_assembly: FALSE

# further details to configure the way Ceedling handles release code
:release_build:
  :output: MyApp.out
  :use_assembly: FALSE
  :artifacts: []

# Plugins are optional Ceedling features which can be enabled. Ceedling supports
# a variety of plugins which may effect the way things are compiled, reported, 
# or may provide new command options. Refer to the readme in each plugin for 
# details on how to use it.
:plugins:
  :load_paths: []
  :enabled:
    # - beep                           # beeps when finished, so you don't waste time waiting for ceedling
    - module_generator                # handy for quickly creating source, header, and test templates
    #- gcov                           # test coverage using gcov. Requires gcc, gcov, and a coverage analyzer like gcovr
    #- bullseye                       # test coverage using bullseye. Requires bullseye for your platform
    #- command_hooks                  # write custom actions to be called at different points during the build process
    #- compile_commands_json_db       # generate a compile_commands.json file
    #- dependencies                   # automatically fetch 3rd party libraries, etc.
    #- subprojects                    # managing builds and test for static libraries
    - fff        # use FFF instead of CMock

    # Report options (You'll want to choose one stdout option, but may choose multiple stored options if desired)
    #- report_build_warnings_log
    #- report_tests_gtestlike_stdout
    #- report_tests_ide_stdout
    #- report_tests_log_factory
    - report_tests_pretty_stdout
    #- report_tests_raw_output_log
    #- report_tests_teamcity_stdout

# Specify which reports you'd like from the log factory
:report_tests_log_factory:
  :reports:
    - json 
    - junit 
    - cppunit 
    - html 

# override the default extensions for your system and toolchain
:extension:
  #:header: .h
  #:source: .c
  #:assembly: .s
  #:dependencies: .d
  #:object: .o
  :executable: .out
  #:testpass: .pass
  #:testfail: .fail
  #:subprojects: .a

# This is where Ceedling should look for your source and test files.
# see documentation for the many options for specifying this.
:paths:
  :test:   

  :source:
    - src/Common/src/**

  :include:
    - src
    - ra_cfg/**
    - ra_gen/**
    - ra/**
    - src/Common/**
    - src/Debug/**

  :support:
    - test/support
  :libraries: []

# You can even specify specific files to add or remove from your test
# and release collections. Usually it's better to use paths and let
# Ceedling do the work for you!
:files:
  :test: []
  :source: []

# Compilation symbols to be injected into builds
# See documentation for advanced options:
#  - Test name matchers for different symbols per test executable build
#  - Referencing symbols in multiple lists using advanced YAML
#  - Specifiying symbols used during test preprocessing
:defines:
  :test:
    :*:
    - TEST # Simple list option to add symbol 'TEST' to compilation of all files in all test executables
    - LOG_DISABLED
    #- _TARGET_THREADX_
    #- _TARGET_RENESAS_RA_
    #- _TARGET_WINDOWS_
    #- INC_WINDOWS_H=\"windows.h\" 
  
  :release: []

  # Enable to inject name of a test as a unique compilation symbol into its respective executable build. 
  :use_test_definition: FALSE 

# Configure additional command line flags provided to tools used in each build step
# :flags:
#   :release:
#     :compile:         # Add '-Wall' and '--02' to compilation of all files in release target
#       - -Wall
#       - --O2
#   :test:
#     :compile:
#       '(_|-)special': # Add '-pedantic' to compilation of all files in all test executables with '_special' or '-special' in their names
#         - -pedantic
#       '*':            # Add '-foo' to compilation of all files in all test executables
#         - -foo

# Configuration Options specific to CMock. See CMock docs for details
:cmock:
  # Core conffiguration
  :plugins:                        # What plugins should be used by CMock?
    - :ignore
    - :callback
  :verbosity:  2                   # the options being 0 errors only, 1 warnings and errors, 2 normal info, 3 verbose
  :when_no_prototypes:  :warn      # the options being :ignore, :warn, or :erro

  # File configuration
  :skeleton_path:  ''              # Subdirectory to store stubs when generated (default: '')
  :mock_prefix:  'mock_'           # Prefix to append to filenames for mocks
  :mock_suffix:  ''                # Suffix to append to filenames for mocks

  # Parser configuration
  :strippables:  ['(?:__attribute__\s*\([ (]*.*?[ )]*\)+)']
  :attributes:
     - __ramfunc
     - __irq
     - __fiq
     - register
     - extern

  :c_calling_conventions:
     - __stdcall
     - __cdecl
     - __fastcall
  :treat_externs:  :exclude        # the options being :include or :exclud
  :treat_inlines:  :exclude        # the options being :include or :exclud

  # Type handling configuration
  #:unity_helper_path: ''          # specify a string of where to find a unity_helper.h file to discover custom type assertions
  :treat_as:                       # optionally add additional types to map custom types
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8

  #:treat_as_array:  {}            # hint to cmock that these types are pointers to something
  #:treat_as_void:  []             # hint to cmock that these types are actually aliases of void
  :memcmp_if_unknown:  true        # allow cmock to use the memory comparison assertions for unknown types
  :when_ptr:  :compare_data        # hint to cmock how to handle pointers in general, the options being :compare_ptr, :compare_data, or :smart

  # Mock generation configuration
  :weak:  ''                       # Symbol to use to declare weak functions
  :enforce_strict_ordering: true   # Do we want cmock to enforce ordering of all function calls?
  :fail_on_unexpected_calls: false  # Do we want cmock to fail when it encounters a function call that wasn't expected?
  :callback_include_count: true    # Do we want cmock to include the number of calls to this callback, when using callbacks?
  :callback_after_arg_check: false # Do we want cmock to enforce an argument check first when using a callback?
  #:includes: []                   # You can add additional includes here, or specify the location with the options below
  #:includes_h_pre_orig_header: [] 
  #:includes_h_post_orig_header: []
  #:includes_c_pre_header:  []
  #:includes_c_post_header:  []
  #:array_size_type:  []            # Specify a type or types that should be used for array lengths
  #:array_size_name:  'size|len'    # Specify a name or names that CMock might automatically recognize as the length of an array
  :exclude_setjmp_h:  false        # Don't use setjmp when running CMock. Note that this might result in late reporting or out-of-order failures.

# Configuration options specific to Unity. 
:unity:
  :defines:
    - UNITY_EXCLUDE_FLOAT

# You can optionally have ceedling create environment variables for you before
# performing the rest of its tasks.
:environment: []
# :environment:
#   # List enforces order allowing later to reference earlier with inline Ruby substitution
#   - :var1: value
#   - :var2: another value
#   - :path:            # Special PATH handling with platform-specific path separators
#     - #{ENV['PATH']}  # Environment variables can use inline Ruby substitution
#     - /another/path/to/include

# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

################################################################
# PLUGIN CONFIGURATION
################################################################

# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :utilities:
    - gcovr           # Use gcovr to create the specified reports (default).
    #- ReportGenerator # Use ReportGenerator to create the specified reports.
  :reports: # Specify one or more reports to generate.
    # Make an HTML summary report.
    - HtmlBasic
    # - HtmlDetailed
    # - Text
    # - Cobertura
    # - SonarQube
    # - JSON
    # - HtmlInline
    # - HtmlInlineAzure
    # - HtmlInlineAzureDark
    # - HtmlChart
    # - MHtml
    # - Badges
    # - CsvSummary
    # - Latex
    # - LatexSummary
    # - PngChart
    # - TeamCitySummary
    # - lcov
    # - Xml
    # - XmlSummary
  :gcovr:
    # :html_artifact_filename: TestCoverageReport.html
    # :html_title: Test Coverage Report
    :html_medium_threshold: 75
    :html_high_threshold: 90
    # :html_absolute_paths: TRUE
    # :html_encoding: UTF-8

# :module_generator:
#   :project_root: ./
#   :source_root: source/
#   :inc_root: includes/
#   :test_root: tests/
#   :naming: :snake #options: :bumpy, :camel, :caps, or :snake
#   :includes:
#     :tst: []
#     :src: []
#   :boilerplates:
#     :src: ""
#     :inc: ""
#     :tst: ""

# :dependencies:
#   :libraries:
#     - :name: WolfSSL
#       :source_path:   third_party/wolfssl/source
#       :build_path:    third_party/wolfssl/build
#       :artifact_path: third_party/wolfssl/install
#       :fetch:
#         :method: :zip
#         :source: \\shared_drive\third_party_libs\wolfssl\wolfssl-4.2.0.zip
#       :environment:
#         - CFLAGS+=-DWOLFSSL_DTLS_ALLOW_FUTURE
#       :build:
#         - "autoreconf -i"
#         - "./configure --enable-tls13 --enable-singlethreaded"
#         - make
#         - make install
#       :artifacts:
#         :static_libraries:
#           - lib/wolfssl.a
#         :dynamic_libraries:
#           - lib/wolfssl.so
#         :includes:
#           - include/**

# :subprojects:  
#   :paths:
#    - :name: libprojectA
#      :source:
#        - ./subprojectA/source
#      :include:
#        - ./subprojectA/include
#      :build_root: ./subprojectA/build
#      :defines: []

# :command_hooks:
#   :pre_mock_preprocess:
#   :post_mock_preprocess:
#   :pre_test_preprocess:
#   :post_test_preprocess:
#   :pre_mock_generate:
#   :post_mock_generate:
#   :pre_runner_generate:
#   :post_runner_generate:
#   :pre_compile_execute:
#   :post_compile_execute:
#   :pre_link_execute:
#   :post_link_execute:
#   :pre_test_fixture_execute:
#   :post_test_fixture_execute:
#   :pre_test:
#   :post_test:
#   :pre_release:
#   :post_release:
#   :pre_build:
#   :post_build:
#   :post_error:

################################################################
# TOOLCHAIN CONFIGURATION
################################################################

#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use
# :tools:
#   :test_compiler: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_linker: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_assembler: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_fixture: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_includes_preprocessor: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_file_preprocessor: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_file_preprocessor_directives: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :test_dependencies_generator: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :release_compiler: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :release_linker: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :release_assembler: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE
#   :release_dependencies_generator: 
#     :executable:
#     :arguments: []
#     :name: 
#     :optional: FALSE

:tools_test_compiler:
   :arguments:
   - -Werror=incompatible-pointer-types
   - -ffunction-sections
...

Feb 14 '25 10:02 JoeLowtech

@JoeLowtech -- unfortunately, with v1.0.2, I'm no longer able to reproduce the issue that you're seeing. It works well with the example project, and any of the projects I've attempted thus far. If you find anything that gives a clue as to what might be causing it, I'm definitely interested!

Feb 17 '25 22:02 mvandervoord

@mvandervoord ok, thank you for looking into it. Do you have any suggestions on where I could look for that clue?

At the moment I am using 1.0.0, but hopefully I will be able to update in the future :)

Feb 19 '25 07:02 JoeLowtech

We are seeing this problem with version

1.0.0,
1.0.1 and
the latest 1.0.2 pre-release.

Compling, linking and running test executables is done single threaded, while prepeocessing seems to be in parallel.

Mar 12 '25 12:03 swaldhoer

@swaldhoer -- you're seeing it single-threading in all those places in 1.0.2 also? The bug that was previously causing this during compilation and linking has been fixed and I'm only seeing it in test execution now. Do you mind sharing your project settings and environment info? Perhaps there is a clue as to where the remaining issue is? You are welcome to email them directly to me if you don't want it public (I have a gmail account with the exact same name as my username here)

Mar 12 '25 13:03 mvandervoord

Hi! I found this after I noticed an increase in my test times with v1.0.1 (test:all took 4m34s) . I installed 1.0.2-75cc24b and it dropped to 1m32! So at least in my side, v1.0.2 is significantly faster.

Mar 13 '25 16:03 mundodisco8

@swaldhoer -- you're seeing it single-threading in all those places in 1.0.2 also? The bug that was previously causing this during compilation and linking has been fixed and I'm only seeing it in test execution now. Do you mind sharing your project settings and environment info? Perhaps there is a clue as to where the remaining issue is? You are welcome to email them directly to me if you don't want it public (I have a gmail account with the exact same name as my username here)

I only did a quick check (as you know our buid process is a little complex), but it looked like, we really had the same bad performance on all three versions.

I'll try to send you detailed logs and config next week.

Mar 15 '25 18:03 swaldhoer

@swaldhoer , @mundodisco8 -- are you able to post the :plugins section of your project.yml file? I have a strong suspicion that it's related to some of the overzealous locking that remains. Much of this is in the plugin handling for reporting.

@JoeLowtech -- you were already kind enough to post yours. You're seeing problems with the latest pre-release, and the only output plugin you have enabled is pretty ?

Also, if you don't mind sharing, what version of Ruby do you have installed?

I'm hoping these clues tell me what needs refactoring yet. :)

Mar 19 '25 18:03 mvandervoord

I have been checking other things. Just to give a bit of background, I was speaking with a colleague over Google Meets, that, for some reason, seems to bring my computer to a crawl (even though I'm using quite a beefy Thinkpad X1 with 64GB of RAM and a 13th gen i7 with 14 cores/20 threads). I was showing him the tests and noticed that test:all took nearly 6m30, when I was expecting more like 2m.

After the call, I tried again and test:all took 4m30, which still was quite a lot. I searched a bit and found this issue. Having said that, after I posted my message, I noticed in the task manager that the CPU clock stayed at around 2GHz while compiling the tests (I expected it to push the cores harder), and after a bit more of reasearch down this rabbit hole, I found that my power plan was set to "energy efficient", which seems to be very good at saving power, but takes, with the latest 1.0.2 release, 2m to run a test:all, while at max performance it only takes 40s (but fan noise is unbearable). "Balanced" power plan it is, then 😆.

TL;DR: moving to 1.0.2 improved my results, but my power settings were also to blame. TL;DR2: even when risking going in a tangent, I'm using a driver for an SPI CAN module (this is the header I mock). A single test that takes 28.73s to complete spends almost 23 of those seconds compiling the mock for that header (here's a log v102kg.txt)

this is my complete project.yml

---
# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.

:project:
  # how to use ceedling. If you're not sure, leave this as `gem` and `?`
  :which_ceedling: gem
  :ceedling_version: 1.0.0

  # optional features. If you don't need them, keep them turned off for performance
  :use_mocks: TRUE
  :use_test_preprocessor:  ### NOTE: set to false in Ted's
  :use_backtrace: :simple
  :use_decorators: :auto #Decorate Ceedling's output text. Your options are :auto, :all, or :none

  # tweak the way ceedling handles automatic tasks
  :build_root: test/build
  :test_file_prefix: test_
  :default_tasks:
    - test:all

  # performance options. If your tools start giving mysterious errors, consider
  # dropping this to 1 to force single-tasking
  :test_threads: 10
  :compile_threads: 10

  # enable release build (more details in release_build section below)
  :release_build: FALSE

# Specify where to find mixins and any that should be enabled automatically
:mixins:
  :enabled: []
  :load_paths: []

# further details to configure the way Ceedling handles test code
:test_build:
  :use_assembly: FALSE

# further details to configure the way Ceedling handles release code
:release_build:
  :output: MyApp.out
  :use_assembly: FALSE
  :artifacts: []

# Configure additional command line flags provided to tools used in each build step
:flags:
  :test:
    :compile:
      # '(_|-)special': # Add '-pedantic' to compilation of all files in all test executables with '_special' or '-special' in their names
        # - -pedantic
      '*':            # Add '-foo' to compilation of all files in all test executables
        - -Og
        - -g3
        - -fdiagnostics-color
        - -Wall
        - -Wextra
        # - -Wfatal-errors    # Stop compilation on 1st Error
        # - -foo
        # - --coverage
        # - -ftest-coverage
        # - -lgcov
  # :release:
  #   :compile:         # Add '-Wall' and '--02' to compilation of all files in release target
  #     - -Wall
  #     - --O2

#:test_build:
#  :use_assembly: TRUE

#:release_build:
#  :output: MyApp.out
#  :use_assembly: FALSE

# You can optionally have ceedling create environment variables for you before
# performing the rest of its tasks.
:environment: []
# :environment:
#   # List enforces order allowing later to reference earlier with inline Ruby substitution
#   - :var1: value
#   - :var2: another value
#   - :path:            # Special PATH handling with platform-specific path separators
#     - #{ENV['PATH']}  # Environment variables can use inline Ruby substitution
#     - /another/path/to/include

# override the default extensions for your system and toolchain
:extension:
  #:header: .h
  #:source: .c
  #:assembly: .s
  #:dependencies: .d
  #:object: .o
  :executable: .out
  #:testpass: .pass
  #:testfail: .fail
  #:subprojects: .a

:paths:
  :test:
    - +:test/*
    - -:test/support
  :source:
    - Core/Src
    - Drivers/CMSIS/Include
    - Middlewares/ST/STM32_WPAN/interface/patterns/ble_thread
    - Middlewares/ST/STM32_WPAN/ble/core

  :include:
    - Core/Inc
    - Core/Src
    - Drivers/CMSIS/**
    - Drivers/STM32WBxx_HAL_Driver/Inc
    - Middlewares/Third_Party/FreeRTOS/Source/**
    - Middlewares/ST/STM32_WPAN/**
    - STM32_WPAN/App/**
    - Utilities/lpm/tiny_lpm
  :support:
    - test/support
  :libraries: []

# You can even specify specific files to add or remove from your test
# and release collections. Usually it's better to use paths and let
# Ceedling do the work for you!
:files:
  :test: []
  :source: []

# Compilation symbols to be injected into builds
# See documentation for advanced options:
#  - Test name matchers for different symbols per test executable build
#  - Referencing symbols in multiple lists using advanced YAML
#  - Specifiying symbols used during test preprocessing
:defines:
  :test:
    - TEST # Simple list option to add symbol 'TEST' to compilation of all files in all test executables
    - CEEDLING
    - DEBUG
    - STM32WB55xx
    - USE_HAL_DRIVER
    - USE_FULL_ASSERT
    - USE_MINIMUM_DEMAND_CURRENT
    - THERMAL_MODE_REQUIRED
  :test_kingin:
    - __CC_ARM
  :release: []

  # Enable to inject name of a test as a unique compilation symbol into its respective executable build.
  :use_test_definition: FALSE

# Configuration Options specific to CMock. See CMock docs for details
:cmock:
  # Core conffiguration
  :plugins:                        # What plugins should be used by CMock?
    - :ignore
    - :callback
    - :expect_any_args
    - :ignore_arg
    - :return_thru_ptr
  :verbosity:  2                   # the options being 0 errors only, 1 warnings and errors, 2 normal info, 3 verbose
  :when_no_prototypes:  :warn      # the options being :ignore, :warn, or :erro

  # File configuration
  :skeleton_path:  ''              # Subdirectory to store stubs when generated (default: '')
  :mock_prefix:  'mock_'           # Prefix to append to filenames for mocks
  :mock_suffix:  ''                # Suffix to append to filenames for mocks

  # Parser configuration
  :strippables:  ['(?:__attribute__\s*\([ (]*.*?[ )]*\)+)']
  :attributes:
     - __ramfunc
     - __irq
     - __fiq
     - register
     - extern
  :c_calling_conventions:
     - __stdcall
     - __cdecl
     - __fastcall
  # :treat_externs:  :exclude        # the options being :include or :exclud
  :treat_externs: :include
  :includes:
    - <app_common.h>
    - <skarper_includes.h>
  :treat_inlines:  :exclude        # the options being :include or :exclud

  # Type handling configuration
  #:unity_helper_path: ''          # specify a string of where to find a unity_helper.h file to discover custom type assertions
  :treat_as:                       # optionally add additional types to map custom types
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8
  #:treat_as_array:  {}            # hint to cmock that these types are pointers to something
  #:treat_as_void:  []             # hint to cmock that these types are actually aliases of void
  :memcmp_if_unknown:  true        # allow cmock to use the memory comparison assertions for unknown types
  :when_ptr:  :compare_data        # hint to cmock how to handle pointers in general, the options being :compare_ptr, :compare_data, or :smart

  # Mock generation configuration
  :weak:  ''                       # Symbol to use to declare weak functions
  :enforce_strict_ordering: true   # Do we want cmock to enforce ordering of all function calls?
  :fail_on_unexpected_calls: true  # Do we want cmock to fail when it encounters a function call that wasn't expected?
  :callback_include_count: true    # Do we want cmock to include the number of calls to this callback, when using callbacks?
  :callback_after_arg_check: false # Do we want cmock to enforce an argument check first when using a callback?
  #:includes: []                   # You can add additional includes here, or specify the location with the options below
  #:includes_h_pre_orig_header: []
  #:includes_h_post_orig_header: []
  #:includes_c_pre_header:  []
  #:includes_c_post_header:  []
  #:array_size_type:  []            # Specify a type or types that should be used for array lengths
  #:array_size_name:  'size|len'    # Specify a name or names that CMock might automatically recognize as the length of an array
  :exclude_setjmp_h:  false        # Don't use setjmp when running CMock. Note that this might result in late reporting or out-of-order failures.

# Configuration options specific to Unity.
:unity:
  :defines:
    # - UNITY_EXCLUDE_FLOAT

# You can optionally have ceedling create environment variables for you before
# performing the rest of its tasks.
:environment: []
# :environment:
#   # List enforces order allowing later to reference earlier with inline Ruby substitution
#   - :var1: value
#   - :var2: another value
#   - :path:            # Special PATH handling with platform-specific path separators
#     - #{ENV['PATH']}  # Environment variables can use inline Ruby substitution
#     - /another/path/to/include

# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :html_report: true
  :utilities:
    - gcovr
  :reports:
    - HtmlBasic
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90
    :html_artifact_filename: artefacts.html


#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use


# Plugins are optional Ceedling features which can be enabled. Ceedling supports
# a variety of plugins which may effect the way things are compiled, reported,
# or may provide new command options. Refer to the readme in each plugin for
# details on how to use it.
:plugins:
  :load_paths: []
  :enabled:
    #- beep                           # beeps when finished, so you don't waste time waiting for ceedling
    - module_generator                # handy for quickly creating source, header, and test templates
    - gcov                           # test coverage using gcov. Requires gcc, gcov, and a coverage analyzer like gcovr
    #- bullseye                       # test coverage using bullseye. Requires bullseye for your platform
    #- command_hooks                  # write custom actions to be called at different points during the build process
    #- compile_commands_json_db       # generate a compile_commands.json file
    #- dependencies                   # automatically fetch 3rd party libraries, etc.
    #- subprojects                    # managing builds and test for static libraries
    #- fake_function_framework        # use FFF instead of CMock

    # Report options (You'll want to choose one stdout option, but may choose multiple stored options if desired)
    #- report_build_warnings_log
    #- report_tests_gtestlike_stdout
    #- report_tests_ide_stdout
    #- report_tests_log_factory
    - report_tests_pretty_stdout
    #- report_tests_raw_output_log
    #- report_tests_teamcity_stdout

# Specify which reports you'd like from the log factory
:report_tests_log_factory:
  :reports:
    - json
    # - junit
    # - cppunit
    - html
################################################################
# PLUGIN CONFIGURATION
################################################################

# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :utilities:
    - gcovr           # Use gcovr to create the specified reports (default).
    - ReportGenerator # Use ReportGenerator to create the specified reports.
  :reports: # Specify one or more reports to generate.
    # Make an HTML summary report.
    # - HtmlBasic
    # - HtmlDetailed
    # - Text
    # - Cobertura
    # - SonarQube
    # - JSON
    - HtmlInline
    # - HtmlInlineAzure
    # - HtmlInlineAzureDark
    # - HtmlChart
    # - MHtml
    # - Badges
    # - CsvSummary
    # - Latex
    # - LatexSummary
    # - PngChart
    # - TeamCitySummary
    # - lcov
    # - Xml
    # - XmlSummary
  :gcovr:
    :html_artifact_filename: TestCoverageReport.html
    # :html_title: Test Coverage Report
    :html_medium_threshold: 75
    :html_high_threshold: 90
    # :html_absolute_paths: TRUE
    # :html_encoding: UTF-8
    :report_include: "../Core/Src/*.*"
  :report_generator:
    :history_directory: test/CoverageHistory
...

Mar 19 '25 20:03 mundodisco8

@mundodisco8 -- Thank you! I really appreciate all the datapoints! I suspect this will help!

Mar 19 '25 21:03 mvandervoord

@mvandervoord I switched to a new workstation. This gave me the opportunity to check if it could be a hardware/OS issue. tl;dr: using ruby_installer + msys2 devkit compared to wsl2 32 secs vs 9 seconds!

tl;dr: compilation could still be singe thread in wsl2 but hard to spot.

My new workstation is still a windows system but I transfered my software developement into wsl2. I compared ruby_installer for windows with the wsl2 setup by using ceedling temp_sensor example.

The performance is a lot faster with wsl 2 : 32 secs with ruby_installer vs 9 secs with wsl2 setup.

And connecting the power adapter to the laptop brings compilation down to 5.91 seconds with wsl 2 :D

My own tests are also executed a lot faster now. (45 secs before vs 10 secs now)

With that in mind, the performance issue could be related to the ruby_installer OR the ruby_installer made the performance issue visible. Because the performance is so slow, you can actually "see" that the compilation is done by one single thread, which might be hard to spot if you are using a linux system.

@mundodisco8 Form your post I assume you are also using a windows machine. If you have the opportunity, you could try wsl2 to get a better performance.

Mar 20 '25 07:03 JoeLowtech

This is a known issue with Windows and printing to terminal and not really related to Ruby or multi-processing. Hence we internally on our build system have an option for Windows users to remove printing to terminal (which is single-threaded) and we noticed 70% speed increase (so not quite on Linux level, but when we are talking about seconds/minutes still a lot) with whole process when you print less. I assume wsl2 uses some kind of Linux virtualization which is much faster than Windows terminal.

One more reason to use Linux.

Mar 20 '25 08:03 Letme

@JoeLowtech thanks for the suggestion. I do work in Windows, and my project lives in the NTFS partition of my disk. WSL2 has bad perfomance in that scenario (there's an issue #https://github.com/microsoft/WSL/issues/4197 if you are curious, opened in 2019 and it seems it's unlikely it will be fixed anytime soon). I could copy the code to my WSL "disk" or an ext partition, but it would kill my workflow.

Mar 20 '25 10:03 mundodisco8

@mvandervoord Unfortunatly I have a problem installing the latest prerelease: ERROR: Could not find a valid gem 'unicode-display_width' (~> 3.1) (required by 'C:\Users\johannes.maier\Downloads\ceedling-1.0.2-75cc24b.gem' (>= 0)) in any repository

But the unicode-display_width gem is listed when gem list

Do you know how I can fix this?

Mar 22 '25 09:03 JoeLowtech

@mvandervoord Unfortunatly I have a problem installing the latest prerelease: ERROR: Could not find a valid gem 'unicode-display_width' (~> 3.1) (required by 'C:\Users\johannes.maier\Downloads\ceedling-1.0.2-75cc24b.gem' (>= 0)) in any repository

But the unicode-display_width gem is listed when gem list

Do you know how I can fix this?

On which Ruby version does this happen? I installed this gem yesterday on Ruby 3.4 on Windows without any issue.

Mar 22 '25 09:03 swaldhoer

@mvandervoord Unfortunatly I have a problem installing the latest prerelease: ERROR: Could not find a valid gem 'unicode-display_width' (~> 3.1) (required by 'C:\Users\johannes.maier\Downloads\ceedling-1.0.2-75cc24b.gem' (>= 0)) in any repository But the unicode-display_width gem is listed when gem list Do you know how I can fix this?

On which Ruby version does this happen? I installed this gem yesterday on Ruby 3.4 on Windows without any issue.

@swaldhoer I figured it out. It was an admin rights issue. Starting ruby cmd shell with admin did let met install the newest version.

@mvandervoord Yes the only output plugin I am using is pretty. Deactivating it has no effect on the performance as far as I can tell.

Mar 23 '25 18:03 JoeLowtech

Any updates on this issue?

Sep 02 '25 13:09 skast96

Thus far, it appears the only solutions we have are adding to documentation. It's the OS's issue that we're stuck with. Depending on which OS you're running, you're going to want to search for disabling the first-time validation of new exe's. This will get rid of the problems, but then your OS will stop scanning downloaded files, etc before running them. It's not ideal.

Sep 02 '25 13:09 mvandervoord