Possible issue with partially covered code - Simplecov setup

[luke-hill]

I run Simplecov on site_prism - https://github.com/site-prism/site_prism/pull/33 - This is an area I'm trying to update the coverage % value. For (possibly poor reasons), some code we only unit test and some code we feature test.

• Include the SimpleCov version you are running in your report. 0.17.x
• Include your ruby -e "puts RUBY_DESCRIPTION". ruby -e "puts RUBY_DESCRIPTION" ruby: No such file or directory -- 2.3.8p459 (LoadError) (But I'm using ruby 2.3 on rvm succesfully)

The link referred has failing outputs. I would like to get the PR's to pass Travis.

The coverage % are as follows

RSpec - 98.4%
Cucumber - 95%
RSpec + Cucumber 99.2%

It could be I've set something up incorrect. But I've followed most of the instructions and tried a few different things. Esentially I want simplecov to only make the check for coverage once, at the very end of the command ran. But it seems to make the check after each rubocop / rspec / cucumber command. I'm not sure why.

[PragTob]

@luke-hill :wave: hi and thanks for the issue report!

How do you run your tests? Usually simplecov checks the coverage "at exit" aka when the testing processes finish (this and the merging). I believe I usually solved this with having one take task run all the tests so that when it finishes, it generates the coverage.

See for instance our default task where we run rubocop, rspec and cucumber all in one: https://github.com/colszowka/simplecov/blob/master/Rakefile#L32

[luke-hill]

https://github.com/site-prism/site_prism/blob/master/Rakefile

[PragTob]

The mentioned PR is merged so I don't think this is a current issue anymore.

@luke-hill if there are still problems please let me know.

[luke-hill]

We still have this "hacky" override in our system. Whereby we test for the lowest bound. So I don't think it's fixed.

I can try make the change again. But I'm not sure it will work.

If you want me to leave this closed I can do, but another ticket would be identical.

[PragTob]

@luke-hill

Can you then please try to phrase the problem again. With the PR merged and everything I don't quite get it. What I understand/think is the following:

• you wanna upgrade the minimum_coverage to reflect the current coverage
• this fails (?) because it aborts your test suite after running rspec but you want it to only measure against rspec + cucumber ?

[luke-hill]

That's correct @PragTob Yes.

So atm my coverage is set to 98.4 - Because that's what rspec covers. But rspec + cucumber together covers almost 100%, so I want to set the value to the higher one.

I'll try it again. But this hasn't changed massively our end. So I think it still fails.

[deivid-rodriguez]

The way I go about this is to use the manual report merging feature, by only enforcing a minimum_coverage after reports have been merged.

So I have a rake task like this:

namespace :coverage do
  desc "Merges all result sets into a single coverage report"
    task :collate do
      ENV["COV_COLLATION"] = "true"

      require "simplecov"

      SimpleCov.collate Dir["coverage/*.json"]
    end
end

and then in the .simplecov file do this

SimpleCov.start

SimpleCov.minimum_coverage 99.2 if ENV["COV_COLLATION"]

[luke-hill]

FYI @PragTob I just did this

# in .simplecov I set the following value
SimpleCov.minimum_coverage 99.1

# Then I cd'd into the project directory
$ bundle exec rake
# lots of lines.....
# lots of lines.....
# lots of lines.....
Finished in 3.29 seconds (files took 0.67888 seconds to load)
308 examples, 0 failures

Randomized with seed 37546

Coverage report generated for RSpec to /home/luke/Code/site_prism/coverage. 1678 / 1702 LOC (98.59%) covered.
Line coverage (98.58%) is below the expected minimum coverage (99.10%).
SimpleCov failed with exit 2
rake aborted!
Specs failed

[deivid-rodriguez]

@PragTob Can you try make the last test suite you run (cucumber I guess) set an environment variable so that when reading .simplecov we know when a minimum coverage needs to be enforced? Something similar to what I wrote about but if you're relying on automerged reports, you don't need to mess with coverage colattion at all.

So, something like

# Rakefile

task "cucumber" do
  sh("ENFORCE_MIN_COVERAGE=true cucumber")
end

# .simplecov

SimpleCov.minimum_coverage 99.1 if ENV["ENFORCE_MIN_COVERAGE"]

Without some external information, I don't think simplecov can do anything about this.

[deivid-rodriguez]

Or even better, remove SimpleCov.minimum_coverage 99.1 from your .simplecov file and add it after all your test tasks run?

[deivid-rodriguez]

Actually my last suggestion wouldn't work, but we could provide a SimpleCov.minimum_coverage! 99.1 to do what I meant.

[PragTob]

I think Deivid meant @luke-hill :)

Also thanks for your help Deivid. I'm wondering if we can do anything about it by ourselves, I suspect not no but we could add your excellent workaround suggestion to the README.

There might be some black magic we could pull but I'd have to look at how rake handles things... not sure.

[deivid-rodriguez]

Yeah @PragTob, sorry for the mention, not sure how that happened :sweat_smile:.

I think we could easily provide SimpleCov.minimum_coverage! that forces computing the result and exiting if minimum coverage if not reached.

[luke-hill]

Hang on. I'm confused.

I thought the SimpleCov.minimum_coverage! var was what controlled everything, or am I misunderstood? I.e. this value should be what value we want covering? And I thought using the .simplecov file was what is best advised? Why am I having to hack around it? Am I doing something wrong? This feels like a bug?

[deivid-rodriguez]

The SimpleCov.minimum_coverage! method doesn't exist, it's just a proposal I'm making to make your use case easier.

The current SimpleCov.minium_coverage configures a minimum coverage for simplecov to enforce when your test process exits. Since you have several different test processes, that doesn't really work for you.

So I'm proposing SimpleCov.minimum_coverage! that you manually call after all your test processes are done, and all your coverage reports are merged, and that exits with an error status if the final report doesn't meet your minimum coverage criteria.

[dmolesUC]

I have a similar issue. In my tests:

1. I have two RSpec Rake tasks, spec:rails and spec:standalone, each of which runs a separate set of tests with separate helpers. They need to be run separately because they have different load paths (one tests with Rails present, one without).
2. I have a top-level spec task that calls both: task spec: ['spec:rails', 'spec:standalone']

As for SimpleCov:

1. I have a coverage task that sets ENV['COVERAGE'] ||= 'true' and then calls Rake::Task[:spec].invoke.
2. Each set of tests has its own _helper.rb, so rails_helper.rb sets SimpleCov.command_name('spec:rails') and standalone_helper.rb sets SimpleCov.command_name('spec:standalone').
3. Each set, by itself, only covers about 90% of the code (the remainder being Rails-specific or no-Rails-specific).
4. In .simplecov, I have minimum_coverage set to 100%.

When I run the coverage task, it fails after the first group, because of course it's less than 100% coverage, and the task exits.

If I redefine the top-level task to use multitask and run the two groups in parallel, the task still fails (returns a nonzero exit code), but both sets of tests get run and the final collated report shows 100% coverage.

Is there any way to delay the application of minimum_coverage till the collation of the final result?

@deivid-rodriguez when/where do your tests actually get executed and the separate .resultset.json files get generated?

[luke-hill]

@deivid-rodriguez I never did the original env var thing (Because I didn't quite follow / agree), but your bang method seems nice and clean (And could live in the configuration file)

Did we make any progress on this?

[deivid-rodriguez]

Nop.

[luke-hill]

Hi there. I'm working again on some stuff and hitting this issue. Any news?