simplecov-ruby
Different code percentage across environments with same config
Hi, congrats for the great work here! I'm trying to setup my CI in a Bitbucket repo and to set a minimum code coverage using Simplecov. My issue is that the coverage is different between environments (local machine vs CI run on remote) even with same configs.
Local machine file paths extracted from coverage.json
app/controllers/application_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/helpers/application_helper.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",
config/application.rb",
config/boot.rb",
config/environment.rb",
config/environments/test.rb",
config/initializers/assets.rb",
config/initializers/content_security_policy.rb",
config/initializers/devise.rb",
config/initializers/filter_parameter_logging.rb",
config/initializers/inflections.rb",
config/initializers/permissions_policy.rb",
config/initializers/rolify.rb",
config/routes.rb",
spec/controllers/leads_controller_spec.rb",
spec/controllers/users/omniauth_callbacks_controller_spec.rb",
spec/factories/leads.rb",
spec/factories/roles.rb",
spec/factories/users.rb",
spec/models/lead_spec.rb",
spec/models/role_spec.rb",
spec/models/user_spec.rb",
spec/support/database_cleaner.rb",
spec/support/helpers/auth_helper.rb",
Remote CI file paths (extracted from coverage.json):
app/channels/application_cable/channel.rb",
app/channels/application_cable/connection.rb",
app/controllers/application_controller.rb",
app/controllers/graphql_controller.rb",
app/controllers/home_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/pages/hello_world_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/controllers/users/sessions_controller.rb",
app/graphql/marteleiro_schema.rb",
app/graphql/mutations/base_mutation.rb",
app/graphql/resolvers/base_resolver.rb",
app/graphql/types/base_argument.rb",
app/graphql/types/base_connection.rb",
app/graphql/types/base_edge.rb",
app/graphql/types/base_enum.rb",
app/graphql/types/base_field.rb",
app/graphql/types/base_input_object.rb",
app/graphql/types/base_interface.rb",
app/graphql/types/base_object.rb",
app/graphql/types/base_scalar.rb",
app/graphql/types/base_union.rb",
app/graphql/types/mutation_type.rb",
app/graphql/types/node_type.rb",
app/graphql/types/query_type.rb",
app/helpers/application_helper.rb",
app/jobs/application_job.rb",
app/mailers/application_mailer.rb",
app/models/ability.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",
config/application.rb",
config/boot.rb",
config/environment.rb",
config/environments/test.rb",
config/initializers/assets.rb",
config/initializers/content_security_policy.rb",
config/initializers/devise.rb",
config/initializers/filter_parameter_logging.rb",
config/initializers/inflections.rb",
config/initializers/permissions_policy.rb",
config/initializers/rolify.rb",
config/routes.rb",
spec/controllers/leads_controller_spec.rb",
spec/controllers/users/omniauth_callbacks_controller_spec.rb",
spec/factories/leads.rb",
spec/factories/roles.rb",
spec/factories/users.rb",
spec/models/lead_spec.rb",
spec/models/role_spec.rb",
spec/models/user_spec.rb",
spec/support/database_cleaner.rb",
spec/support/helpers/auth_helper.rb",
The coverage results are:
- Local: 98.18%
- CI Remote: 92.62%
To fix it I tried adding some filters to exclude the following paths:
# /spec_helper.rb
require 'simplecov'
require 'simplecov-json'
module SimpleCov
module Formatter
class MergedFormatter
def format(result)
SimpleCov::Formatter::HTMLFormatter.new.format(result)
SimpleCov::Formatter::JSONFormatter.new.format(result)
end
end
end
end
SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
SimpleCov.start 'rails' do
add_filter 'config/'
add_filter 'tmp/'
add_filter 'public/'
add_filter 'lib/'
add_filter 'vendor/'
add_filter 'storage/'
add_filter 'db/'
add_filter 'bin/'
end
...
But after that, the result was weirder:
app/channels/application_cable/channel.rb",
app/channels/application_cable/connection.rb",
app/controllers/application_controller.rb",
app/controllers/graphql_controller.rb",
app/controllers/home_controller.rb",
app/controllers/leads_controller.rb",
app/controllers/pages/hello_world_controller.rb",
app/controllers/users/omniauth_callbacks_controller.rb",
app/controllers/users/sessions_controller.rb",
app/graphql/marteleiro_schema.rb",
app/graphql/mutations/base_mutation.rb",
app/graphql/resolvers/base_resolver.rb",
app/graphql/types/base_argument.rb",
app/graphql/types/base_connection.rb",
app/graphql/types/base_edge.rb",
app/graphql/types/base_enum.rb",
app/graphql/types/base_field.rb",
app/graphql/types/base_input_object.rb",
app/graphql/types/base_interface.rb",
app/graphql/types/base_object.rb",
app/graphql/types/base_scalar.rb",
app/graphql/types/base_union.rb",
app/graphql/types/mutation_type.rb",
app/graphql/types/node_type.rb",
app/graphql/types/query_type.rb",
app/helpers/application_helper.rb",
app/jobs/application_job.rb",
app/mailers/application_mailer.rb",
app/models/ability.rb",
app/models/application_record.rb",
app/models/lead.rb",
app/models/role.rb",
app/models/user.rb",
With a coverage of 18.88%
Currently my SimpleCov config is set at the beginning of the spec_helper.rb file but I also tried using .simplecov and had the same issues.
Dependencies' versions:
ruby (3.3.0)
rails (7.2.1)
rspec (3.13.0)
simplecov (0.22.0)
simplecov_json_formatter (0.1.4)
No, sir @rikas I just put it aside to check it later in the project since it's a project that I'm working solo. If you find the solution, please share with us.
Do you have config.eager_load = ENV["CI"].present? in your config/environments/test.rb, by any chance?
Newer versions of Rails add this automatically, and it changes the load order in CI versus on your local machine, which could impact what SimpleCov is able to cover.
