Simplecov cannot generate a new coverage report because it does not skip writing already existing asset files.

[chase-stevens]

Hi all,

I was able to generate a coverage/index.html the first time I ran rspec with simplecov. Each subsequent time, I have encountered an error where, when attempting to create the coverage/index.html file, simplecov gets an error that it is not allowed to open an asset file.

1: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1292:in `open' /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1292:in `initialize': Permission denied @ rb_sysopen - /home/chase/eic/coverage/assets/0.10.2/smoothness/images/ui-bg_glass_65_ffffff_1x400.png (Errno::EACCES)

What we believe to be happening is that simplecov is attempting to copy files from the gem kept in our nix/store directory and write them to the coverage/assets directory in the application. The first time writing was successful; however, simplecov does not appear to have the necessary logic to skip writing files that already exist.

Full trace below:

Traceback (most recent call last):
	26: from /nix/store/9gf3q3vzp3f25vvknqb8cf3d68wrfnq2-ruby2.5.5-simplecov-0.17.0/lib/ruby/gems/2.5.0/gems/simplecov-0.17.0/lib/simplecov/defaults.rb:29:in `block in <top (required)>'
	25: from /nix/store/9gf3q3vzp3f25vvknqb8cf3d68wrfnq2-ruby2.5.5-simplecov-0.17.0/lib/ruby/gems/2.5.0/gems/simplecov-0.17.0/lib/simplecov.rb:201:in `run_exit_tasks!'
	24: from /nix/store/9gf3q3vzp3f25vvknqb8cf3d68wrfnq2-ruby2.5.5-simplecov-0.17.0/lib/ruby/gems/2.5.0/gems/simplecov-0.17.0/lib/simplecov/configuration.rb:182:in `block in at_exit'
	23: from /nix/store/9gf3q3vzp3f25vvknqb8cf3d68wrfnq2-ruby2.5.5-simplecov-0.17.0/lib/ruby/gems/2.5.0/gems/simplecov-0.17.0/lib/simplecov/result.rb:48:in `format!'
	22: from /nix/store/jws2anzzrgimgd4v95b05niw2j4q2rrg-ruby2.5.5-simplecov-html-0.10.2/lib/ruby/gems/2.5.0/gems/simplecov-html-0.10.2/lib/simplecov-html.rb:18:in `format'
	21: from /nix/store/jws2anzzrgimgd4v95b05niw2j4q2rrg-ruby2.5.5-simplecov-html-0.10.2/lib/ruby/gems/2.5.0/gems/simplecov-html-0.10.2/lib/simplecov-html.rb:18:in `each'
	20: from /nix/store/jws2anzzrgimgd4v95b05niw2j4q2rrg-ruby2.5.5-simplecov-html-0.10.2/lib/ruby/gems/2.5.0/gems/simplecov-html-0.10.2/lib/simplecov-html.rb:19:in `block in format'
	19: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:392:in `cp_r'
	18: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1461:in `fu_each_src_dest'
	17: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1477:in `fu_each_src_dest0'
	16: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1463:in `block in fu_each_src_dest'
	15: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:393:in `block in cp_r'
	14: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:415:in `copy_entry'
	13: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1392:in `wrap_traverse'
	12: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1392:in `each'
	11: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1393:in `block in wrap_traverse'
	10: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1392:in `wrap_traverse'
	 9: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1392:in `each'
	 8: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1393:in `block in wrap_traverse'
	 7: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1390:in `wrap_traverse'
	 6: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:418:in `block in copy_entry'
	 5: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1259:in `copy'
	 4: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1291:in `copy_file'
	 3: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1291:in `open'
	 2: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1292:in `block in copy_file'
	 1: from /nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1292:in `open'
/nix/store/fnmw699zb2pk8gn38ic99vx9ax53n9g6-ruby-2.5.5/lib/ruby/2.5.0/fileutils.rb:1292:in `initialize': Permission denied @ rb_sysopen - /home/chase/eic/coverage/assets/0.10.2/smoothness/images/ui-bg_glass_65_ffffff_1x400.png (Errno::EACCES)

I am using RSpec for tests and I run my tests using the following command nix-shell --run "rspec"

I am requiring and staring SimpleCov in my spec_helper.rb as seen below. Additionally, I have been able to successfully run simplecov once before, so I do not think this is the issue.

require 'simplecov'
SimpleCov.start 'rails'

Software Versions

• Ruby 2.5.5p157 (2019-03-15) [x86_64-linux]
• Rails v5.1.6
• Simplecov (0.17.0)
• Simplecov-html (0.10.2)

Please let me know if you have any questions or if I can provide any additional information and I'll be happy to help.

Best, Chase

[PragTob]

Hi there @chase-stevens,

thanks for the report :green_heart:

The interesting question for me would be why it doesn't have access to these files any more? Seems like there might be something wrong there?

The reason it writes already existing asset files again is that the gem version and hence the files might have changed. Sure, we could include some hash in the file names or even the version names but that'd require a build process etc... as the time to write the files again isn't too big we just write them new on each run.

I'm also not sure if it'd totally solve your problem. My fear is that if the access rights for the asset files aren't given then simplecov probably also doesn't have the rights to override coverage/index.html so we wouldn't win anything there.

So - I don't think there's much we can do here and it seems more a problem with your file permissions. If you think we can help please let us know :)

[PragTob]

@chase-stevens :wave: hi there, any update on this and the questions from your side? :)

[PragTob]

Closing due to missing feedback

[felixscheinost]

When using Nix all the gems are stored read-only in /nix/store.

As you can see simplecov-html does a simple FileUtils.cp_r which preserves rights.

I would propose that the fix would be to chmod the files, e.g. 644?

[PragTob]

@felixscheinost what is "Nix" ?

I'm not sure whether this is another problem we should fix or that users should fix... like we already integrate with so much. Us messing with chmoding prodocues potential further issues (what if we can't chmod...) so I'm almost inclined to say that this would be user level to chmod them themselves.

[felixscheinost]

Sorry, forgot to explain what Nix is - seems like I am too much in my own filter bubble :)

Nix is a build tool and package manager that focuses on purity and reproducibility. All packages are stored in /nix and include the hash of the build inputs and outputs in the file names (see the error log in the initial post). Files generated by the build are 444 so that they can’t be changed after the hash is calculated.

I'll try to find a user level solution and will get back to you later.

[PragTob]

@felixscheinost have you found a user level solution yet?

While I generally would like to have things working everywhere, doing this would introduce some overhead. Of course, if we switched the asset compilation on simplecov-html to something more modern, have it easy to attach a hash to the file name then it'd be easy but right now I think there are other issues deserving our attention.

[emptyflask]

I'm a NixOS user too, and I've worked around this by adding the following to test_helper.rb:

require 'simplecov'
Pathname.new(__FILE__).join('..','..','coverage').tap do |cov|
  # clear old coverage results
  FileUtils.rm_rf cov if cov.exist?
end
SimpleCov.start('rails')

But if simplecov skipped rewriting the assets, that would be great!

[PragTob]

I was just thinking, we might introduce asset inlining which would also solve this problem - right?

[slinkp]

Would it hurt anything if simplecov did @emptyflask 's workaround? rm the asset tree if it exists, immediately before writing new ones?

[PragTob]

@slinkp maybe :shrug:

Couldn't do it like this because it deletes the whole path. People might (accidentally) throw files in there and we can't just delete them all.

Could we delete all the files we're gonna copy over? Probably. Next thing I know maybe for some reason under some OS we don't have the right to create files there... and then that case starts failing all of a sudden.

So, I'm not very inclined to include it into default. Asset inlining and/or having hashes build style at the end of the assets would be the proper solutions I think. For the time being people can implement this workaround.

Reopening to know/remember it'd be nice to fix these.

[PragTob]

Hey,

thanks to feedback here I thought about it some more and theoretically I guess we could:

• read the files and just write them as new files to avoid this problem (as they should have no different rights set then)
• we could catch the errors and warn but continue, that seems incredibly specific and probably not good though

As there seem to be a bit more people running into this than I expected I might try to provide a fix with higher priority, but no promises time is a bit volatile these days :)

[abathur]

I ran into this yesterday (by way of bashcov, which I naively thought would be trivial to package 😆 for an experiment I'm doing on fuzzing shell scripts...).

I took 3 swings at local workarounds with varying success; outlining them in case it helps anyone.

1. My first approach was to try and get simplecov using a distinct directory for every run. I eventually got this working, but this was a bad first-choice hampered by the fact that that there's a limited env-var API, and bashcov doesn't appear to pass unrecognized command-line args on to simplecov (less relevant here, but I had to patch in something like SimpleCov.coverage_dir ENV.fetch('BASHCOV_COVERAGE_DIR', 'coverage')).

   I solved this by generating a shell wrapper that called mktemp -d to create a new coverage directory on every invocation. I guess this breaks auto-merging coverage. I didn't take any extra time with this one to look at how to combine it with the simplecov collect/merge flow; the complexity of hunting down the different directories seems to be worse than just re-using the command-name to make a subdir for each set of output.

2. Probably the biggest hack, but also most-successful approach: I generated a wrapper that runs chmod -R 755 ./coverage || true before each invocation to forcibly enable writing.

3. The smallest intervention that got around the basic error here was specifying the SimpleFormatter. I don't see a way to set the formatter outside of the Ruby API, so I patched SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter into bashcov. I'm not certain this worked correctly; I didn't see an error message, but I just have a large .resultset.json and .resultset.json.lock file. This avoids the file permission issue, at least. If the resultset.json really is the the output to expect here, it's obvious that I'd have to go find or write some additional processing tool to extract useful information from it.

[PragTob]

@abathur regarding 3. - interesting, I'd hope that bashcov would allow for this to be set/not use the html formatter if they don't need it. .resultset.json has most of the data you need, it lacks a lot of post processing though and is considered an internal file/format whose structure can change without notice. We basically dump the ruby coverage data in there and use it as a local synchronization method across processes.

[abathur]

@PragTob Ok. I assumed it was just the internal state file (and that for some reason specifying the formatter did disable the default, but not produce an intentional output) but good to confirm.

Here's my patched bashcov just in case the context helps (or in case it's dead-obvious what I'm doing wrong, but don't feel like you need to debug it, especially if you think this is a bashcov problem).

$ cat /nix/store/ykfcfk0airk1l3ipsxgs872scdddshjv-ruby2.6.6-bashcov-1.8.2/lib/ruby/gems/2.6.0/gems/bashcov-1.8.2/bin/bashcov
Wed Jul 29 2020 10:09:40 --> 
#!/nix/store/34x64aql2bmlffjf6jzqrbk05h9y5nxr-ruby-2.6.6/bin/ruby
# frozen_string_literal: true

lib = File.expand_path("../../lib", __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)

require "bashcov"

Bashcov.parse_options! ARGV

runner = Bashcov::Runner.new Bashcov.command
status = runner.run
coverage = runner.result

require "simplecov"

SimpleCov.start

SimpleCov.command_name Bashcov.command_name
SimpleCov.root Bashcov.root_directory
SimpleCov.coverage_dir ENV.fetch('BASHCOV_COVERAGE_DIR', 'coverage44')
SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter

result = SimpleCov::Result.new(coverage)
if SimpleCov.use_merging
  SimpleCov::ResultMerger.store_result(result)
  result = SimpleCov::ResultMerger.merged_result
end

SimpleCov.at_exit do
  puts "Run completed using #{Bashcov.fullname}"
  result.format!
end

exit status.exitstatus