Coverage changes after each run

[Minituff]

Hello, I'm new to using Ruby so I apologized if this is a simple question, but every time I run my tests my coverage\index.html is giving me a different output.

The strange thing is, the output is correct and consistent within my Github Actions pipeline.

Im using: bashcov 3.1.1 with Bash 5.0, Ruby 3.2.2, and SimpleCov 0.21.2

My `.simplecov` file

require 'simplecov'
require 'simplecov-cobertura'
require "simplecov-html"

# frozen_string_literal: true

# SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter  # Converts the `.resultset.json` to `coverage.xml`

# SimpleCov::Formatter::JSONFormatter, # This formatter breaks the build

# Use multiple formatters
SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
  SimpleCov::Formatter::SimpleFormatter,
  SimpleCov::Formatter::CoberturaFormatter,
  SimpleCov::Formatter::HTMLFormatter,
])

# .simplecov
SimpleCov.start 'rails' do
  command_name 'Unit Tests'
  enable_coverage :branch
  primary_coverage :branch
  add_filter %r{^/snippets/}
  add_filter %r{^/.git/}
  add_filter %r{^/tests/}
  add_filter "pkg/test.sh"
  add_group "Pkg scripts", "/pkg"

  enable_coverage_for_eval # Must be at the bottom and Must be here, even though it throws a 'command not found' error
end

Here is the relevant part of my Gitlab CI workflow

    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: head
    
    - name: Install Ruby dependencies
      run: |
        gem install bashcov simplecov-cobertura simplecov-html
    
    - name: Run unit tests
      run: bashcov ./tests/tests.sh && ls -la coverage

    - name: Upload reports to Codecov
      uses: codecov/codecov-action@v4-beta
      # uses: codecov/codecov-action@v3
      env: # Required for v4
        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
      with:
        fail_ci_if_error: true
        files: coverage/coverage.xml # The json file will not work here

This is the output when I run the tests locally changes each time, but ranges from 50-65%

This is the output generated from my testing pipeline. It's always accurate and consistent (~90%)

Are you able to help me determine why the coverage is not the same when run codecov locally?

How to replicate the issue

1. Clone my repo git clone [email protected]:Minituff/nautical-backup.git
2. Install dependencies: gem install bashcov simplecov-cobertura simplecov-html
3. Run the tests bashcov ./tests/tests.sh && ls -la coverage

My repo is open source so feel free to look around if I did not include something https://github.com/Minituff/nautical-backup

Thanks!

[jarthod]

Just a wild guess but I had a similar issue cause by spring locally, because it was using a forked process. So in my case I run it with bundle exec rspec instead of bin/rspec to avoid the binstub when I need coverage and it works better, you can also try the suggestions here: https://github.com/simplecov-ruby/simplecov#want-to-use-spring-with-simplecov