CircleCI: invalid byte sequence in UTF-8 (ArgumentError) / make sure we can work with Unicode characters in file names

[joshuamcginnis]

I'm getting the following error only when running simplecov in a CircleCI environment. The same command works fine on our local environments.

I'm using ruby 2.5.0 and simplecov 0.15.1.

# /spec/spec_helper.rb
require 'simplecov'

# save to CircleCI's artifacts directory if we're on CircleCI
if ENV['CIRCLE_ARTIFACTS']
  dir = File.join(ENV['CIRCLE_ARTIFACTS'], 'coverage')
  SimpleCov.coverage_dir(dir)
end

SimpleCov.start

Failure Message

Finished in 0.36838 seconds (files took 3.18 seconds to load)
32 examples, 0 failures

Traceback (most recent call last):
	11: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/defaults.rb:68:in `block in <top (required)>'
	10: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/configuration.rb:180:in `block in at_exit'
	 9: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov.rb:82:in `result'
	 8: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov.rb:63:in `add_not_loaded_files'
	 7: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov.rb:63:in `each'
	 6: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov.rb:66:in `block in add_not_loaded_files'
	 5: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:20:in `classify'
	 4: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:20:in `map'
	 3: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:20:in `each'
	 2: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:20:in `foreach'
	 1: from /home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:21:in `block in classify'
/home/ubuntu/darwin/vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/lib/simplecov/lines_classifier.rb:21:in `=~': invalid byte sequence in UTF-8 (ArgumentError)

bundle exec rspec --require spec_helper spec --format documentation returned exit code 1

Do you have any ideas on how to appropriately troubleshoot this error?

[thesmart]

I'm getting this same error and have noticed that it has something to do with UTF-8 characters in some source files. I think simplecov is opening files in ascii text mode?

[PragTob]

:wave:

Hi, I think the offending line is this one.

if line =~ self.class.no_cov_line

So, something is wrong with that line or how simplecov is. Printing out the line that fails would be helpful and would help us create a test case either for line classifier or more integrationy including the file loading to make that pass.

[joshuamcginnis]

I monkey-patched simplecov in spec_helper so that I could print the line and additional information that may or may not be useful.

Outpt

"LINE: # localized to Espa\xF1ol thus:\n" #\xF1 is unicode for n + ~
"MATCH: (?-mix:^(\\s*)#(\\s*)(\\:nocov\\:))"
"LANG: en_US.UTF-8"
"encoding_internal: UTF-8"
"encoding_external: UTF-8"

The exception is being triggered by https://github.com/colszowka/simplecov/blob/master/spec/fixtures/iso-8859.rb#L2, which is:

# localized to Español thus:

The offending file is:

Edit

The offending file is: vendor/bundle/ruby/2.5.0/gems/simplecov-0.15.1/spec/fixtures/iso-8859.rb

Great, so why I don't we just ignore this file. Unfortunately, that doesn't work either (add_filter "vendor" inside of the start block had no effect). And I wonder why is this file being covered anyway?

Isn't the default bundler_filter supposed to filter out files in /vendor/bundler?

[joshuamcginnis]

While this doesn't address the issue of the ñ, I realized that by activating the built-in rails profile, the issue goes away as it doesn't cover the offending file in the vendor folder:

Simplecov.start 'rails'

[PragTob]

Hi there,

add_filter should work in general I don't know why it wouldn't. Unless you use something like track_files or so that overrides the filtering iirc. An example repo would be nice but afaik it works?

We probably also shouldn't bundle up our spec fixtures so removing those might also help :D

[PragTob]

Updated to highlight what one of the root causes here seems to be