Surprise behavior with Kotlin's upcoming annotation-default-target changes

[ZacSweers]

Prior reading:

• https://youtrack.jetbrains.com/issue/KT-73255
• https://github.com/Kotlin/KEEP/blob/change-defaulting-rule/proposals/change-defaulting-rule.md

In short, kotlin is changing how annotations on properties are generated into bytecode, and I found some surprising behavior with how this interacts with this library.

Consider this test:

@RunWith(TestParameterInjector::class)
class ExampleTest(@TestParameter private val isEnabled: Boolean)

Prior to this behavior, the resulting java bytecode is roughly

@RunWith(TestParameterInjector.class)
public class ExampleTest {
  private final boolean isEnabled;

  public ExampleTest(@TestParameter boolean isEnabled) {
    this.isEnabled = isEnabled;
  }
}

When this test is run, it creates two scenarios ([isEnabled=true,isEnabled=false])

If you opt into this new behavior however, such as passing -Xannotation-default-target=param-property to the compiler, the resulting bytecode adds the annotation to both the field and the parameter.

@RunWith(TestParameterInjector.class)
public class ExampleTest {
  @TestParameter
  private final boolean isEnabled;

  public ExampleTest(@TestParameter boolean isEnabled) {
    this.isEnabled = isEnabled;
  }
}

This breaks the runner in a humorous and deliciously horrible way: it now believes there are two inputs, and it creates a matrix of both! The result is now four tests for all the permutations of the inputs 😅.

This creates a pathological case where the field is set to a different value than the input parameter in one of these test scenarios, wherein the test fails with what looks like a nonsensical test input state.

I'm not sure what the right solution is to this in the library itself (dedupe by name?), but filing now for awareness!

Some workarounds for consumers:

• Use the @param:TestParameter site target to prevent it from getting applied to the field.
• Don't have a backing field for the parameter

[nymanjens]

Thanks @ZacSweers for the very detailed description!

The combination of a final field annotated with @TestParameter (kind of surprised that this even can work because I don't recall ever testing this on final fields) and a constructor parameter with the same name should allow us to safely ignore the field.

In Java, I know that compiling parameter names into the bytecode is often an opt-in option. That then makes comparison harder. Not sure how this is for Kotlin. I can imagine it's always compiled into the bytecode to make things like named arguments work.

[yogurtearl]

For additional context, this new Kotlin behavior is not currently enabled by default in the latest stable compiler (2.1.20), but will be the default in some not to far out future version of Kotlin (looks like maybe in Kotlin 2.3?).

It would be nice if this was fixed before that future behavior goes stable.

[ln-12]

I think that I just ran into this issue after updating to Kotlin 2.2.0 and trying to find out which library is causing my issue.

I am using this library to generate snapshot images on Android similar to how it is shown in this example. Before the update, the test description contained the name of the preview that should be generated. Now it contains an array of two values which matches the matrix that is described here. As I have over 850 tests that are using this setup, it now generates more than 700.000 test runs 🫠

Is there any news about when this will be fixed? Both workarounds from above work for me but it took me way to long to find the root cause.

[yogurtearl]

If all your tests are in Kotlin https://github.com/cashapp/burst is a potential alternative.

[ln-12]

If I remember correctly I tried that one before and ran into some issue which was a blocker for my project. But maybe it's worth giving it a try again.

Found it: this use case is not supported by Burst at the moment (see https://github.com/cashapp/burst/issues/65).

[sergio-sastre]

@ln-12 have you tried with the standard ParameterizedTestRunner from Junit? That is also an alternative for your use case and can also be used with Paparazzi tests.

However, I am unsure whether the same issue would also arise

[ln-12]

@sergio-sastre Yes, that's also an option that works for me 👍🏻

[nymanjens]

It would be nice if this was fixed before that future behavior goes stable.

Sorry. I put this on my TODO list, but not high enough. I'll look into this shortly.

[nymanjens]

Update: I was able to reproduce the problem.

I've also noticed that the parameter names (exposed in Java) are optional for the latest Kotlin compiler, so that makes the most straightforward heuristic (look at the parameter name vs the field name) at least only a partial solution.

[nymanjens]

This problem should now be fixed in version 1.19:

• If you have javaParameters set to true in the Kotlin compiler, this problem is fixed
• If you have javaParameters set to false and there are fields and constructor parameters with overlapping types you get an error message with (hopefully) clear and straightforward instructions for how to fix the error