Incompatibility with rspec --options on 6.0+

[irphilli]

I'm using https://github.com/sj26/rspec_junit_formatter, and I am passing the following options when running tests:

-r rspec_junit_formatter
--format RspecJunitFormatter
--out tmp/rspec/junit.xml

This is supposed to produce formatted output to tmp/rspec/junit.xml. This works fine with knapsack_pro 5.7.0. As soon as I update to 6.x, the output file is no longer being created.

[ArturT]

Hi @irphilli

I can't reproduce the issue. I tested this locally and it seems to work for Queue Mode and Regular Mode.

Can you share the Knapsack Pro command that you use to run tests?

Do you use this configuration here? https://docs.knapsackpro.com/ruby/rspec/#formatters-rspec_junit_formatter-json

[irphilli]

Thanks for looking into it! I'm using something similar with a file for the options with: knapsack_pro:rspec[--options ./.rspec]

I also have some custom reporters that might be part of the issue, but the non-custom part of the config is:

-r rspec_junit_formatter
--format RspecJunitFormatter
--out tmp/rspec/junit.xml
--format documentation

I'll take a look at some of the custom reporters I have if it can't be reproduced.

[irphilli]

Looking further into the test output, it looks like --format documentation isn't working as expected either! So, it seems like maybe something is off with using --options.

[ArturT]

What is your RSpec version?

I created a script to run Knapsack Pro with the RSpec --options provided, and the JUnit XML file is generated correctly. https://github.com/KnapsackPro/rails-app-with-knapsack_pro/commit/675a1188f39cf8a67ffb447a4487db50fcf98619

I'm not sure how to reproduce the issue. Can you share more details about your config? What's the content of rails_helper.rb/spec_helper.rb. What are your custom formatters. The full list of options that are provided to the Knapsack Pro command.

If that's private data, you can share it with the support email https://knapsackpro.com/contact

You can also look at recent changes we did in 6.x version of the knapsack_pro gem. We added a custom RSpec formatter to track tests execution time. Maybe this is causing some conflict with your formatters.

• CHANGELOG https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#600

• the new formatter added https://github.com/KnapsackPro/knapsack_pro-ruby/pull/229/files#diff-224867ff67d486513c374ab2291d9d98db5a02b567424f0e282dcc90988df458R1

[irphilli]

Here are my versions:

RSpec 3.12
  - rspec-core 3.12.2
  - rspec-expectations 3.12.3
  - rspec-mocks 3.12.6
  - rspec-rails 6.1.0
  - rspec-support 3.12.1

Seems like maybe just something up with my particular setup specifically (will dig into that more). I've trimmed down the options file to:

--format documentation

And, the output is not coming out in the "documentation" format, when run like: knapsack_pro:rspec[--options ./.rspec]. However, when run like knapsack_pro:rspec[--format documentation], it outputs as expected.

[ArturT]

If the --format documentation is present in the ./.rspec file then the output should be in the documentation format. Is --format documentation inside of the ./.rspec file?

This is how I use options: https://github.com/KnapsackPro/rails-app-with-knapsack_pro/blob/1f217df18c71db7dae01f28f89a0e394b33a2a61/bin/knapsack_pro_queue_rspec_junit_with_rspec_custom_options#L18

Maybe you should use .rspec instead of ./.rspec. Why do you add ./?

Please try bundle exec rake "knapsack_pro:rspec[--options .rspec]". Please remember to add "to wrap the options passed to the knapsack_pro command. Otherwise the options won't work.

[ArturT]

@irphilli I got an email notification that you were asking about:

Looking at the command generated by the runner:

/usr/local/bin/ruby -I/app/vendor/bundle/ruby/3.2.0/gems/rspec-core-3.12.2/lib:/app/vendor/bundle/ruby/3.2.0/gems/rspec-support-3.12.1/lib /app/vendor/bundle/ruby/3.2.0/gems/rspec-core-3.12.2/exe/rspec --format documentation --options .rspec --format progress --format KnapsackPro::Formatters::TimeTracker --default-path spec ...

It looks like an extra --format progress is getting inserted (overrides --format documentation). This doesn't yet explain my original issue, but seems like there is something different in the generated command with 6.0+ when it comes to the format options.

If a formatter is not detected in Regular Mode then --format progress is added as a default formatter. It seems that --options .rspec is not loaded as the RSpec configuration option, and the --format documentation could not be detected. It's something that we could improve here: https://github.com/KnapsackPro/knapsack_pro-ruby/pull/229/files#diff-23b86e488780bfc91dda40439ac5e4f6be1c67e0739bad6c286d36bae402adf3R54

Do you see a similar issue for Queue Mode: bundle exec rake "knapsack_pro:queue:rspec"?

[irphilli]

I will try queue mode! I was going to follow up with some additional info, but you got to the same conclusion :).

[irphilli]

So, I do see the same issue in queue mode, but looks like it was already like that in 5.x.

Going back to standard mode, I went ahead and got rid of our options file (passed in arguments directly), and things are working fine, even with the custom reporters. The only strange thing is that the generated rspec command still has an extra --format progress in there.

[ArturT]

I think it is a bug in our code. The extra --format progress option is added when formatters are empty but we don't check if someone passed --format option, or --options .rspec_custom_file that has --format inside of the file.

It was introduced here for Regular Mode.

I notice also issue for Queue Mode.

I reported this in the internal ticket.

@irphilli Please use inline options. Thanks to that, the --format option is applied.

Example:

# Queue Mode (--format documentation is applied inline)
bundle exec rake "knapsack_pro:queue:rspec[--options .rspec_custom.options --format RspecJunitFormatter --out tmp/rspec.xml --format documentation]"

# Regular Mode (--format documentation is applied inline)
bundle exec rake "knapsack_pro:rspec[--options .rspec_custom.options --format RspecJunitFormatter --out tmp/rspec.xml --format documentation]"

[ArturT]

I'm closing this in favor of using inline options as in the above example.