Fails to cover file loaded multiple times

[Aethelflaed]

I've a bot which defines each statement in a separate file to simplify testing.

To test the files, I remove all statements and then just load (not require, as it won't work) the ones I need for the specific test.

The coverage seems to work fine until I reach a test which loads all the statements.
If I had already tested e.g. ping.rb, and then this test runs, it will reload ping.rb anyway and reset all covering information about that file.

The coverage does not merge multiple loads of the same file but instead keeps only the last occurence.

I'm not sure but this may be due to ruby's Coverage and not Simplecov.

ruby 2.0.0p598 (2014-11-13 revision 48408) [x86_64-linux] also tested with ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]

[bf4]

You need to give us more information on how to reproduce this issue, otherwise there is nothing we can do. Please read CONTRIBUTING.md file for more information about creating bug reports. Thanks!

Sample code would be great.

[Aethelflaed]

File.rb:

def method
  puts 'Hello'
end

require 'simplecov'

SimpleCov.start
puts "SimpleCov v#{SimpleCov::VERSION}"

load File.expand_path('../file.rb', __FILE__)
method
load File.expand_path('../file.rb', __FILE__)

Result:

SimpleCov v0.10.0 Hello SimpleCov failed to recognize the test framework and/or suite used. Please specify manually using SimpleCov.command_name 'Unit Tests'. Coverage report generated for Unknown Test Framework to /tmp/test3387/coverage. 1 / 2 LOC (50.0%) covered.

If I comment the second load, I get:

SimpleCov v0.10.0 Hello SimpleCov failed to recognize the test framework and/or suite used. Please specify manually using SimpleCov.command_name 'Unit Tests'. Coverage report generated for Unknown Test Framework to /tmp/test3387/coverage. 2 / 2 LOC (100.0%) covered.

[bf4]

I don't quite follow where you're running the SimpleCov.start from, which file has the method defined, and which file is loading it.

maybe try this https://github.com/colszowka/simplecov/issues/328#issuecomment-68450477:

require 'json'
SimpleCov.at_exit do
  filename = File.join(SimpleCov.root, "coverage/#{$$}.json")
  File.write filename, JSON.pretty_generate(Coverage.result.sort)
end
pwd = File.expand_path('..', __FILE__)
File.write("file_that_defines_method_in_main.rb", <<-EOF)
    puts "File #{__FILE__} loaded by #{caller[0]}"
    def method_in_main
      puts "Hello, I am called from #{__FILE__} #{__LINE__} by #{caller[0]}"
    end
EOF
SimpleCov.start do
  command_name "test run"
end
load File.join(pwd, "file_that_defines_method_in_main.rb")
method_in_main
load File.join(pwd, "file_that_defines_method_in_main.rb")

[Aethelflaed]

Oh sorry, I was running SimpleCov.start from the second file, but even with your test case, the output is different if I comment the second load:

[
  [
    "/tmp/test22731/file_that_defines_method_in_main.rb",
    [
      1,
      1,
      0,
      null
    ]
  ]
]

[
  [
    "/tmp/test22731/file_that_defines_method_in_main.rb",
    [
      1,
      1,
      1,
      null
    ]
  ]
]

The result is the same with ruby 2.0.0 and ruby 2.2.0

[Aethelflaed]

Actually, I can get a correct result by patching Object#load and changing the command_name

def load(file, wrap = false)
  SimpleCov.result
  SimpleCov.start do
    command_name "#{command_name}1"
  end
  Kernel.load(file, wrap)
end

SimpleCov.result apparently merges the result so it's fine, but the default at_exit doesn't seem to call it, so I added:

SimpleCov.at_exit do
  SimpleCov.result.format!
end

I think the problem comes from ruby's Coverage module, not SimpleCov, but we can work around.

Ruby 2.3 introduce Coverage.peek_result which may be used to simplify the workaround

[PragTob]

Yes probably a coverage bug or rather loading a file multiples times isn't supported as per the coverage module and its intended usage.

Might be worth adding as a limitation to the README.

[trisys3]

@Aethelflaed How/where did you patch Object#load in your previous comment? I'm thinking of implementing your solution because I believe I'm having the same issue. Sorry, I know it's been awhile since you last had to think about this.

[Aethelflaed]

@trisys3 I just def load directly in the file as stated in https://github.com/simplecov-ruby/simplecov/issues/389#issuecomment-104863943, this replaces the default Object#load that would get called otherwise