kcov enters infinite loop when run with bats executable

[twils0]

When I run kcov (as detailed below), it enters an infinite loop. I can't seem to get kcov to honor my exclude/include flags. I've tried many different variations of {include|exclude}-{path|pattern}.

I installed executables for kcov and bats in my project. I want to get the coverage for script.sh, based on the bats test script_test.bats.

Here's an example of the commands I'm tried:

kcov/bin/src/kcov --bash-dont-parse-binary-dir --exclude-pattern=bats/ --include-pattern=script.sh coverage/ ./bats/bin/bats script_tests.bats

Here's my project structure:

project/
project/script.sh
project/script_test.bats
project/bats/bin/bat    <= bats executable
project/bats/bats-core
project/bats/bats-support
project/bats/bats-assert
project/kcov/*  <= cloned from Github
project/kcov/bin/src/kcov    <= kcov executable

Here's the last few lines of the infinite loop that kcov enters into. I can't get kcov to ignore these files related to the bats executable.

I've also downloaded bats through Homebrew. When I run kcov against the bats executable from Homebrew, this infinite loop again occurs with paths pointing at the bats Homebrew directory, instead of the bats directory in my project (as shown below).

# kcov@project/bats/lib/bats-core/common.bash@5@printf '# %s\n' 'kcov@project/bats/lib/bats-core/common.bash@5@printf '\''# %s\n'\'' '\''kkcov@project/bats/lib/bats-core/tracing.bash@281@for path in "${BATS_DEBUG_EXCLUDE_PATHS[@]}"'\'''
# kcov@project/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@project/bats/lib/bats-core/common.bash@15@printf '\''%s\n'\'' '\''# kkcov@project/bats/lib/bats-core/tracing.bash@281@for path in "${BATS_DEBUG_EXCLUDE_PATHS[@]}"'\'''
# kcov@project/bats/lib/bats-core/common.bash@14@read -r line
# kcov@project/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@project/bats/lib/bats-core/common.bash@4@IFS='
# kcov@project/bats/lib/bats-core/common.bash@14@read -r line
# kcov@project/bats/lib/bats-core/common.bash@4@IFS=
# kcov@project/bats/lib/bats-core/common.bash@4@read -r line
# kcov@project/bats/lib/bats-core/common.bash@5@printf '# %s\n' 'kcov@project/bats/lib/bats-core/common.bash@15@printf '\''%s\n'\'' '\''# kkcov@project/bats/lib/bats-core/tracing.bash@195@[[ 0 -gt 0 ]]'\'''
# kcov@project/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@project/bats/lib/bats-core/common.bash@14@read -r line'
# kcov@project/bats/lib/bats-core/common.bash@14@read -r line
# kcov@project/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@project/bats/lib/bats-core/common.bash@4@read -r line'
# kcov@project/bats/lib/bats-core/common.bash@14@read -r line
# kcov@project/bats/lib/bats-core/common.bash@4@IFS=
# kcov@project/bats/lib/bats-core/common.bash@4@read -r line
# kcov@project/bats/lib/bats-core/common.bash@5@printf '# %s\n' kcov@project/bats/lib/bats-core/common.bash@4@IFS=
# kcov@project/bats/lib/bats-core/common.bash@4@IFS=
# kcov@project/bats/lib/bats-core/common.bash@4@read -r line
# kcov@project/bats/lib/bats-core/common.bash@5@printf '# %s\n' 'kcov@project/bats/lib/bats-core/common.bash@14@read -r line'
# kcov@project/bats/lib/bats-core/common.bash@4@IFS=
# kcov@project/bats/lib/bats-core/common.bash@4@read -r line
# kcov@project/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@project/bats/lib/bats-core/common.bash@5@printf '\''# %s\n'\'' kcov@project/script_tests.bats@42@DEPLOY_END=2022-01-01T00:00:00Z'
# kcov@project/bats/lib/bats-core/common.bash@14@read -r line
^C# kcov@project/bats/lib/bats-core/common.bash@5@printf '# %s\n' 'kcov@project/bats/lib/bats-core/common.bash@4@read -r line'

[SimonKagstrom]

Strange. Does kcov work with any bash script in your installation?

One thing you can try, although I don't think it will give that much more information, is to run with --debug=31. With some luck, it might indicate where it gets stuck.

[twils0]

Thank you for your fast reply!

My question overall is: why can't I exclude the bats binary (executable) directory and/or the bats directory? I'm using --bash-dont-parse-binary-dir --exclude-pattern=bats/. I've tried countless variations of the include and exclude commands.

If I run kcov against script_cli.sh directly (just a bash script not a bats test obviously), it seems to work. No infinite loop. It prints out some lines of code from the script and logs that look like the following.

0x7f9bb29041b0 REPORT hit at 0x1b4098079bf
0x7f9bb2804080 REPORT hit at 0x1b5aa9424c3
0x7f9bb29041b0 REPORT hit at 0x1b5098079bf

I ran with "--debug=31". I'll post the repeating logs from the end of kcov's output below. The logs seem different, but I'm not sure if they will help. From what I can tell, it looks like kcov prints some logs related to script_cli.sh and script_cli_test.bats at the beginning, and it then starts printing out logs related to bats - common.bash, tracing.bash, etc. - in an infinite loop.

I'm sure that it's related to the bats executable that I pass to kcov.

• If I pass the bats executable from inside my project, the paths from the infinite loop point to my project's bats-core directory (logs shown below with the path: "project/bats/lib/bats-core/").
• If I pass the bats executable I installed from Homebrew, the paths from the infinite loop point to the Homebrew bats-core directory (the same logs with the path" "/usr/local/Cellar/bats-core").

# kkcov@project/bats/lib/bats-core/tracing.bash@17@local limit=6
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i = 2 ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-support/src/output.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=batslib_is_single_line
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-support/src/output.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=batslib_print_kv_single_or_multi
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-assert/src/assert.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=assert_equal
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/script_cli_tests.bats
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=test_Missing_DEPLOY-2d5fMETHOD
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@298@bats_emit_trace
# kkcov@project/bats/lib/bats-core/tracing.bash@195@[[ 0 -gt 0 ]]
# kkcov@project/bats/bats-support/src/output.bash@82@bats_debug_trap project/bats/bats-support/src/output.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@278@local NORMALIZED_INPUT
# kkcov@project/bats/lib/bats-core/tracing.bash@279@bats_normalize_windows_dir_path NORMALIZED_INPUT project/bats/bats-support/src
# kkcov@project/bats/lib/bats-core/tracing.bash@161@local output_var=NORMALIZED_INPUT path=project/bats/bats-support/src
# kkcov@project/bats/lib/bats-core/tracing.bash@162@[[ NORMALIZED_INPUT != NORMALIZED_INPUT ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@165@[[ project/bats/bats-support/src == ?:* ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@171@NORMALIZED_INPUT=project/bats/bats-support/src
# kkcov@project/bats/lib/bats-core/tracing.bash@173@printf -v NORMALIZED_INPUT %s project/bats/bats-support/src
# kkcov@project/bats/lib/bats-core/tracing.bash@280@local path
# kkcov@project/bats/lib/bats-core/tracing.bash@281@for path in "${BATS_DEBUG_EXCLUDE_PATHS[@]}"
# kkcov@project/bats/lib/bats-core/tracing.bash@282@[[ project/bats/bats-support/src == /\p\r\o\j\e\c\t/\b\a\t\s\/\l\i\b\/* ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@281@for path in "${BATS_DEBUG_EXCLUDE_PATHS[@]}"
# kkcov@project/bats/lib/bats-core/tracing.bash@282@[[ project/bats/bats-support/src == /\p\r\o\j\e\c\t/\b\a\t\s\/\l\i\b\/* ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@281@for path in "${BATS_DEBUG_EXCLUDE_PATHS[@]}"
# kkcov@project/bats/lib/bats-core/tracing.bash@282@[[ project/bats/bats-support/src == /\p\r\o\j\e\c\t/\b\a\t\s\/\l\i\b\/* ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@290@[[ NOTSET == NOTSET ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@290@[[ NOTSET == NOTSET ]]
# kkcov@project/bats/lib/bats-core/tracing.bash@293@BATS_DEBUG_LASTLAST_STACK_TRACE=(${BATS_DEBUG_LAST_STACK_TRACE[@]+"${BATS_DEBUG_LAST_STACK_TRACE[@]}"})
# kkcov@project/bats/lib/bats-core/tracing.bash@295@BATS_DEBUG_LAST_LINENO=(${BASH_LINENO[@]+"${BASH_LINENO[@]}"})
# kkcov@project/bats/lib/bats-core/tracing.bash@296@BATS_DEBUG_LAST_SOURCE=(${BASH_SOURCE[@]+"${BASH_SOURCE[@]}"})
# kkcov@project/bats/lib/bats-core/tracing.bash@297@bats_capture_stack_trace
# kkcov@project/bats/lib/bats-core/tracing.bash@12@local test_file
# kkcov@project/bats/lib/bats-core/tracing.bash@13@local funcname
# kkcov@project/bats/lib/bats-core/tracing.bash@14@local i
# kkcov@project/bats/lib/bats-core/tracing.bash@16@BATS_DEBUG_LAST_STACK_TRACE=()
# kkcov@project/bats/lib/bats-core/tracing.bash@17@local limit=6
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i = 2 ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-support/src/output.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=batslib_is_single_line
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-support/src/output.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=batslib_print_kv_single_or_multi
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/bats/bats-assert/src/assert.bash
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=assert_equal
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@22@test_file=project/script_cli_tests.bats
# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=test_Missing_DEPLOY-2d5fMETHOD
# kkcov@project/bats/lib/bats-core/tracing.bash@24@BATS_DEBUG_LAST_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( ++i ))
# kkcov@project/bats/lib/bats-core/tracing.bash@19@(( i < limit  ))
# kkcov@project/bats/lib/bats-core/tracing.bash@298@bats_emit_trace
^C# kkcov@project/bats/lib/bats-core/tracing.bash@195@[[ 0 -gt 0 ]]

[twils0]

I can tell one thing from the logs above. This line references the first test in my bats test file:

# kkcov@project/bats/lib/bats-core/tracing.bash@23@funcname=test_Missing_DEPLOY-2d5fMETHOD

My first test is called: Missing DEPLOY_METHOD. It prints out error messages, if my script is missing the DEPLOY_METHOD environment variable. The test provides all required environment variables, except DEPLOY_METHOD, and looks for the correct error messages from the script output.

[sschmid]

Hi! I found this issue and I might have run into the same issue. I created a minimal example repo to reproduce the issue: https://github.com/sschmid/kcov-sample It's 1 bash script to test called src/app and 1 bats test in test/app.bats, including a GitHub action to check the logs.

The project structure is as follows:

kcov-sample
├── src
│   └── app
└── test
    ├── bats
    ├── test_helper
    └── app.bats

The test succeeds as expected using this command

test/bats/bin/bats --tap test

However, the test fails and kcov seems to be stuck in an endless loop

kcov --include-path=src ${{ github.workspace }}/coverage test/bats/bin/bats --tap test

see GitHub action workflow file: https://github.com/sschmid/kcov-sample/blob/main/.github/workflows/tests.yaml

see GitHub action results: https://github.com/sschmid/kcov-sample/actions

See Run 37 - Add test coverage with kcov which fails on step "Test Coverage". I cancelled the run, because it seemed to be stuck in an endless loop.

Note: The workflow is configured to run on macOS using runs-on: macos-latest because my real project needs to run on macOS. I have other projects using the kcov/kcov docker container, where everything works as expected.

I hope this helps! Any idea what the problem could be?

[sschmid]

Here's are the logs for the "Test Coverage" step mentioned in my previous comment: 5_Test Coverage.txt

[SimonKagstrom]

@sschmid so do I understand it correct that it's a MacOS-specific problem?

[sschmid]

@SimonKagstrom I don't know.

To test if it's a macOS-only issue, I wanted to change the GitHub action to runs-on: ubuntu-latest, but it seems theres no kcov package that I can install. I tried apt search kcov but there are no matches. Is there a way to install a kcov package on ubuntu?

Please feel free to modify the GitHub action with a PR or let me know what I can change to investigate further: https://github.com/sschmid/kcov-sample/blob/main/.github/workflows/tests.yaml

I have other projects that don't require macOS where I used the kcov/kcov docker container, which works fine.

[sschmid]

@SimonKagstrom Update: it seems to be macOS related

I changed the GitHub action to runs-on: ubuntu-latest and followed the install steps (clone, cmake, make) and I'm building kcov from source as a build step.

See .github/workflows/tests.yaml (happy for feedback) https://github.com/sschmid/kcov-sample/blob/main/.github/workflows/tests.yaml

It works, see job/30119462484 https://github.com/sschmid/kcov-sample/actions/runs/10852741048/job/30119462484

So it seems it's macOS related. I used brew install kcov in the macos runner. I didn't build kcov from source (which I guess I shouldn't in a build step anyway?)

[twils0]

@SimonKagstrom @sschmid I didn't think to mention it above. I'm using macOS as well. Also, I built kcov from source.

It seems likely it's a macOS issue.

[SimonKagstrom]

@sschmid @twils0 good to know!

The homebrew package is updated quickly after each release, so it's the latest available right now, so no need to build by yourselves.

And as for the issue, I don't really know why it would behave differently. One can of course be the bash version, which I guess will be newer with homebrew. Another thing, if you haven't already tried is to use the --bash-method=DEBUG.

Unfortunately, I basically never use kcov for bash scripts myself, although a lot of other people seem to do that. Therefore progress with these bugs are typically slow, at least from my side...

[sschmid]

@SimonKagstrom Update: For completeness I also have a run where I build kcov from sources, see https://github.com/sschmid/kcov-sample/actions/runs/10853960497/job/30123376533

Same issue, stuck in endless loop.

Note: I couldn't get building from source to work by following the steps in INSTALL.md. This is how I was able to build it successfully (disclaimer: no idea what ninja is, ChatGPT helped me here ...)

https://github.com/sschmid/kcov-sample/blob/eed34838145fd57718282ab0ecd0d5c78697f327/.github/workflows/tests.yaml#L24-L32

brew install cmake ninja wget openssl zlib dwarfutils
git clone https://github.com/SimonKagstrom/kcov.git
cd kcov
mkdir build
cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DOPENSSL_ROOT_DIR=$(brew --prefix openssl) ..
ninja

Nice trick and potential addition to INSTALL.md to help setting the openssl path: -DOPENSSL_ROOT_DIR=$(brew --prefix openssl)

[sschmid]

Unfortunately, I basically never use kcov for bash scripts myself, although a lot of other people seem to do that. Therefore progress with these bugs are typically slow, at least from my side...

Understood. If you or anyone else from the community wants to have a look at it, here's a quick summary:

• kcov enters an infinite loop when testing bash scripts with bats on macos-latest
• ubuntu-latest and docker image kcov/kcov work. Likely macOS-only issue
• here's a repo with a minimal setup: 1 source file, 1 bats test, GitHub action running on macos-latest. Feel free to fork, reproduce and investigate
  • repo: https://github.com/sschmid/kcov-sample
  • actions: https://github.com/sschmid/kcov-sample/actions

[SimonKagstrom]

Thanks a lot @sschmid for the investigation!

You didn't happen to try the --bash-method=DEBUG argument to kcov?

And thanks for the documentation update, will fix!

[sschmid]

@SimonKagstrom

Results when using --bash-method=DEBUG: The GitHub action doesn't fail anymore, however coverage is 0%

Code covered:	NaN% 

Here's the run, see step "Test Coverage", no kcov related output in the logs: https://github.com/sschmid/kcov-sample/actions/runs/10883185515/job/30195770019

0% coverage, see generated "kcov-sample Coverage Report", added as an artifact for download: https://github.com/sschmid/kcov-sample/actions/runs/10883185515/artifacts/1937236741

For reference, here's how I invoke kcov in the workflow file, I hope I'm not missing anything: https://github.com/sschmid/kcov-sample/actions/runs/10883185515/workflow#L26-L30

kcov \
  --bash-method=DEBUG \
  --include-path=src \
  ${{ github.workspace }}/coverage \
  test/bats/bin/bats --tap test

[SimonKagstrom]

--configure=bash-force-stderr-input=1 might also be an option to try, although I don't think it will make any difference.

Thanks @sschmid again!

[sschmid]

I thank you!

I gave it a try:

kcov: error: Unknown key bash-force-stderr-input

See this run, step "Test Coverage" https://github.com/sschmid/kcov-sample/actions/runs/10889043350/job/30214821835

Is this correct? Looks like there is sth wrong, it contains 2 =, as in --configure=... and ...input=1.

--configure=bash-force-stderr-input=1

[SimonKagstrom]

Sorry, I'm misrembering :-)

--debug-force-bash-stderr

is the option.

The --configure=X=Y looks strange, but is actually correct (although with not this key, the X value). It's basically a debug-only option.

[sschmid]

Ok thanks! Unfortunately, it's back to infinite loop, see this run, step "Test Coverage"

https://github.com/sschmid/kcov-sample/actions/runs/10889171578/job/30215220875

kcov \
  --debug-force-bash-stderr \
  --include-path=src \
  ${{ github.workspace }}/coverage \
  test/bats/bin/bats --tap test

If there's anything else I can try, let me know. And thanks for your fast replies! Really appreciate it!

[sschmid]

Hi @SimonKagstrom, I see you've tagged a commit with this issue. Let me know if I can help test something!

[SimonKagstrom]

Saw a minor issue, but I don't think it's really related to the hang. If you want to test it, it's on the latest master (i.e., you'll need to build from source).

I also fixed the documentation error you spotted by the way.

[jack-rann]

I, too, ran into this problem running macOS Ventura 13.6.9, but didn't have time to debug it further until now. Starting with the reproduction code from @sschmid, I added the following code at the beginning of src/app:

echo >/dev/tty ">>>>> ${BASH_VERSINFO[*]} <<<<<"
echo >/dev/tty ">>>>> KCOV_BASH_XTRACEFD=${KCOV_BASH_XTRACEFD} <<<<<"
echo >/dev/tty ">>>>> KCOV_BASH_COMMAND=${KCOV_BASH_COMMAND} <<<<<"
echo >/dev/tty ">>>>> BASH_ENV=${BASH_ENV} <<<<<"
echo >/dev/tty ">>>>> BASH_XTRACEFD=${BASH_XTRACEFD} <<<<<"
echo >/dev/tty ">>>>> KCOV_BASH_USE_DEBUG_TRAP=${KCOV_BASH_USE_DEBUG_TRAP} <<<<<"

Executing this, I see the same infinite loop reported above:

$ kcov --include-path=src coverage bats --verbose-run test
>>>>> 5 2 32 1 release x86_64-apple-darwin22.6.0 <<<<<
>>>>> KCOV_BASH_XTRACEFD=2 <<<<<
>>>>> KCOV_BASH_COMMAND=/bin/bash <<<<<
>>>>> BASH_ENV=coverage/bash-helper.sh <<<<<
>>>>> BASH_XTRACEFD=2 <<<<<
>>>>> KCOV_BASH_USE_DEBUG_TRAP= <<<<<
1..1
not ok 1 prints args
# (from function `assert_output' in file /usr/local/lib/bats-assert/src/assert_output.bash, line 194,
#  in test file test/app.bats, line 12)
#   `assert_output "args: test"' failed
# kkcov@/usr/local/Cellar/bats-core/1.11.0/libexec/bats-core/bats-exec-test@353@bats_debug_trap /usr/local/Cellar/bats-core/1.11.0/libexec/bats-core/bats-exec-test
# kkcov@/usr/local/Cellar/bats-core/1.11.0/lib/bats-core/tracing.bash@278@local NORMALIZED_INPUT
# kkcov@/usr/local/Cellar/bats-core/1.11.0/lib/bats-core/tracing.bash@279@bats_normalize_windows_dir_path NORMALIZED_INPUT /usr/local/Cellar/bats-core/1.11.0/libexec/bats-core
# kkcov@/usr/local/Cellar/bats-core/1.11.0/lib/bats-core/tracing.bash@161@local output_var=NORMALIZED_INPUT path=/usr/local/Cellar/bats-core/1.11.0/libexec/bats-core
# kkcov@/usr/local/Cellar/bats-core/1.11.0/lib/bats-core/tracing.bash@162@[[ NORMALIZED_INPUT != NORMALIZED_INPUT ]]
... (interrupted by ctrl-c)

Note that modified src/app reports that the BASH_XTRACEFD file descriptor is 2. So it appears that kcov and bats are co-mingling their trace information on STDERR.

Also note that the modified code also reports KCOV_BASH_COMMAND=/bin/bash. This is the default setting as defined in kcov's src/configuration.cc. macOS Ventura (and I suspect later versions as well) use Bash 3.2:

$ /bin/bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin22)
Copyright (C) 2007 Free Software Foundation, Inc.
$

Instrumenting debug code into src/engines/bash-engine.cc confirms that bashCanHandleXtraceFd() is indeed returning false based on /bin/bash reporting its version as 3.2.

I currently have a newer version of bash installed from HomeBrew:

$ which bash
/usr/local/bin/bash
$ /usr/local/bin/bash --version
GNU bash, version 5.2.32(1)-release (x86_64-apple-darwin22.6.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ 

I can explicitly set the bash executable to use this version of bash, all works as desired:

$ kcov --bash-parser=/usr/local/bin/bash --include-path=src coverage bats --verbose-run test
1..1
>>>>> 5 2 32 1 release x86_64-apple-darwin22.6.0 <<<<<
>>>>> KCOV_BASH_XTRACEFD=64 <<<<<
>>>>> KCOV_BASH_COMMAND=/usr/local/bin/bash <<<<<
>>>>> BASH_ENV=coverage/bash-helper.sh <<<<<
>>>>> BASH_XTRACEFD=64 <<<<<
>>>>> KCOV_BASH_USE_DEBUG_TRAP= <<<<<
ok 1 prints args
$ 

Note that BASH_XTRACEFD is set to 64 (based in the getrlimit() code in src/engines/bash-engine.cc. coverage/index.html now reports coverage of src/app at 90.0% (higher percentage due to the additional statements I added).

I haven't yet tried to integrate this approach into code where I had previously tried to use kcov. I'll do that in the near future and report back.

I think the simple workaround is to force users with older bash implementations to specify --bash-parser=/usr/local/bin/bash (or similar). Alternatively, kcov could attempt to find bash along the user's PATH. It appears that the python interpreter is found in this manner. I did hack together a new path_to_executable() method based on th e existing executable_exists_in_path() method and modified configuration.cc to do setKey("bash-command", path_to_executable("bash"));. This does appear to work, so this approach may be worthwhile considering.

[SimonKagstrom]

Well done @jack-rann ! Sounds like an upcoming PR then :-)

[sschmid]

@jack-rann Awesome! Thanks a lot!

I updated the sample project, seems to work 🤩

I invoke kcov like this now:

kcov \
  --bash-parser="$(which bash)" \
  --include-path=src \
  ${{ github.workspace }}/coverage \
  test/bats/bin/bats --tap test

The run passes https://github.com/sschmid/kcov-sample/actions/runs/10939179540/job/30368882734

The generated coverage report has 75% coverage https://github.com/sschmid/kcov-sample/actions/runs/10939179540/artifacts/1952460476

Super happy that that fixed it!

A note on bash on macOS: Since macOS Catalina (2019), macOS switched default shell from bash to zsh. I doubt macOS will ever ship a newer version than bash 3.2 (we're at bash 5.x now) Therefore I imagine it's common practice to install bash with brew install bash. To repect that in my bash tool I use #!/usr/bin/env bash instead of #!/bin/bash. Maybe kcov could use that as a senseable default too, to make it use the user's preferred bash version.

[sschmid]

Btw, in the meantime I tried to make it work with bashcov. I couldn't make it work (and will keep using kcov), but turns out they have a similar issue with using /bin/bash by default, so I had to do this: --bash-path "$(which bash)"

bashcov --bash-path "$(which bash)" -- test/bats/bin/bats --tap test

bashcov printed similar infos like @jack-rann added by default, so it was obvious and easier to find that --bash-path option. I'm not too familiar with kcov except the standard usage, so maybe you already have this, but it might be useful to dump some of this info like paths and versions as debug info.

[SimonKagstrom]

The fix @jack-rann indicates would do that, i.e., take bash from the environment instead of using /bin/bash hardcoded.

I had forgotten, but apparently we use a different path on FreeBSD as well. It's this part from configuration.cc:

		setKey("python-command", "python");
#ifdef __FreeBSD__
		setKey("bash-command", "/usr/local/bin/bash");
#else
		setKey("bash-command", "/bin/bash");
#endif
		setKey("kernel-coverage-path", "/sys/kernel/debug/kprobe-coverage");

And it's probably enough to just change it to "bash" only.

[sschmid]

I'm happy to report that it also worked testing it on the real project running 125 tests: https://github.com/sschmid/pw-terminal-password-manager/actions/runs/10942688064

@SimonKagstrom Thank you so much for this project! Thanks @jack-rann for finding a fix!

Happy coding!

[sschmid]

@SimonKagstrom Ok cool! Fyi, homebrew used to install into /usr/local/bin too, but starting with Apple M1 silicon, it installs into /opt/homebrew/bin

[jack-rann]

I'm happy to submit a PR @SimonKagstrom, but it will be a couple of days until I can get to that.

I can report that using kcov --bash-parser=$(which bash) in a suite of 132 bats based tests does indeed work. Coverage statistics are correctly generated. However, I am seeing some bleed through of what appears to be set -x style output from bats in the kcov output. I'll try to isolate this behavior and open a new issue for it.

[sschmid]

@jack-rann you can ping me on your new issue if you want, I also have unreadable bats output in some of my projects and can provide logs and help.

[...] Coverage statistics are correctly generated.

I found I have different coverage percentages when comparing same commit ran with GitHub actions, ran locally, and ran with kcov docker container.

I will try to isolate and follow up with a nee issue.

[jack-rann]

@SimonKagstrom, any pointers for running unit tests? After enabling GitHub actions in my fork, both my manually triggered workflow and automatically triggered (based on my pushes to branch 458-bash-path) workflow failed. See https://github.com/jack-rann/kcov/actions. I've also not been successful seeing how to manually run tests in my local, checked out working copy.

Simple bash script and bats test shows desired operation. First invocation uses kcov v43, while second uses locally compiled version. Coverage statistics in the latter are correct given the script and single test implemented.

sample (458-bash-path)$ /usr/local/bin/kcov --dump-summary --include-path=src coverage bats test
1..1
ok 1 prints args
{
  "files": [

  ],
  "percent_covered": "0.00",
  "covered_lines": 0,
  "total_lines": 0
}
sample (458-bash-path)$ 
sample (458-bash-path)$ ../build/src/kcov --dump-summary --include-path=src coverage bats test
1..1
ok 1 prints args
{
  "files": [
    {"file": "/Users/jrann/git/kcov/sample/src/app.sh", "percent_covered": "85.71"}
  ],
  "percent_covered": "85.71",
  "covered_lines": 6,
  "total_lines": 7
}
sample (458-bash-path)$

Code is finished (see https://github.com/SimonKagstrom/kcov/compare/master...jack-rann:kcov:458-bash-path?expand=1), though it would be good to add some tests to validate correctness (as well as validate on platforms other than macOS). If this is sufficient, then I can raise the PR.