Breaking change in 1.12.4: IllegalStateException: Invisible parameter type of ...

[cushon]

Upgrading the version of bytebuddy used by https://github.com/apache/beam from 1.12.3 to 1.12.4 caused a regression, see also https://github.com/apache/beam/issues/22749

I don't see an obvious culprit in the changelog, was this a deliberate change?

I found these issues discussing the same error message, but they don't obviously have the same root cause:

• https://github.com/spockframework/spock/issues/776
• https://issues.apache.org/jira/browse/BEAM-5061

Repro

git clone https://github.com/Talend/beam-samples.git
git checkout 95b9c4aef156af4f5c6f287538a44b6a3adfb4fa

Override bytebuddy version:

$ git diff
diff --git a/pom.xml b/pom.xml
index e004383..e669e47 100644
--- a/pom.xml
+++ b/pom.xml
@@ -97,7 +97,13 @@
     </modules>

     <dependencyManagement>
-        <dependencies>
+      <dependencies>
+
+        <dependency>
+          <groupId>net.bytebuddy</groupId>
+          <artifactId>byte-buddy</artifactId>
+          <version>1.12.4</version>
+        </dependency>

             <!-- SDK -->
             <dependency>

Run the tests:

$ mvn verify -D"beam.version"="2.41.0"

The test fails with

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.952 s <<< FAILURE! - in SerializationTest
[ERROR] SerializationTest.nonSerilizableTest  Time elapsed: 1.925 s  <<< ERROR!
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Invisible parameter type of SerializationTest$1 arg0 for public SerializationTest$1$DoFnInvoker(SerializationTest$1)
Caused by: java.lang.IllegalStateException: Invisible parameter type of SerializationTest$1 arg0 for public SerializationTest$1$DoFnInvoker(SerializationTest$1)

The test passes if the version is downgraded back to 1.12.3.

I used maven 3.6.3 and JDK 11:

$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.13
Default locale: en_US, platform encoding: UTF-8
OS name: "linux"

I think the test is transforming this input: https://github.com/Talend/beam-samples/blob/74b5dfae2b2d1ac4e1272467e7d4584f852b0888/serializableTests/src/test/java/SerializationTest.java#L22

with this use of bytebuddy: https://github.com/apache/beam/blob/cf9ea1f442636f781b9f449e953016bb39622781/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java#L117

[raphw]

This is not a deliberate change. I remember that Kotlin does something with marker types (being package private) that is not normally compilable in Java. Let me find out why this happens, I don't see why this is supposed to happen with the changes that were made.

[raphw]

I do not quite understand what changed yet, but you should always set:

new ByteBuddy().with(TypeValidation.DISABLED) in non-debugging scenarios for performance reason. Without this validation, the invisibility will also be ignored.

[raphw]

I added a dependency management to the test project that forces use of BB 1.12.13 instead and the same error occurs. But there is indeed an inconsistency in Byte Buddy when handling the default package.

[cushon]

Thanks for taking a look!

new ByteBuddy().with(TypeValidation.DISABLED) in non-debugging scenarios for performance reason. Without this validation, the invisibility will also be ignored.

Thanks, I will look in to doing this.

I added a dependency management to the test project that forces use of BB 1.12.13 instead and the same error occurs

Can you expand on this? If I add:

diff --git a/pom.xml b/pom.xml
index e004383..4d15c9f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -99,6 +99,12 @@
     <dependencyManagement>
         <dependencies>

+            <dependency>
+              <groupId>net.bytebuddy</groupId>
+              <artifactId>byte-buddy</artifactId>
+              <version>1.12.3</version>
+            </dependency>
+
             <!-- SDK -->
             <dependency>
                 <groupId>org.apache.beam</groupId>

I only see BB 1.12.3 being resolved, and the tests pass.

$ mvn dependency:tree -D"beam.version"="2.41.0" |& grep bytebuddy
...
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.12.3:compile

[cushon]

I tried setting .with(TypeValidation.DISABLED) everywhere as suggested. I'm no longer getting the java.lang.IllegalStateException: Invisible parameter type ... error from bytebuddy, but now I'm getting a NoSuchMethodException.

So it seems like something else changed in BB 1.12.4 that's causing different code to be generated.

The chnages to use .with(TypeValidation.DISABLED) are in: https://github.com/cushon/beam/compare/master...bytebuddy-typevalidation

Demo:

Rebuild beam:

$ cd ~/src/beam
$ ./gradlew -Ppublishing -PdistMgmtSnapshotsUrl=~/.m2/repository/ publishToMavenLocal -x test

Rerun tests with local build of beam. It fails with a NoSuchMethodException instead of the IllegalStateException from bytebuddy:

$ cd ~/src/beam-samples/
$ mvn  verify -D"beam.version"="2.42.0-SNAPSHOT"
...
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.lang.NoSuchMethodException: SerializationTest$1$DoFnInvoker.<init>(SerializationTest$1)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: SerializationTest$1$DoFnInvoker.<init>(SerializationTest$1)
Caused by: java.lang.NoSuchMethodException: SerializationTest$1$DoFnInvoker.<init>(SerializationTest$1)

[raphw]

Ah, version dot-3, not dot-13. I'll look into that.

[cushon]

I think it bisects to https://github.com/raphw/byte-buddy/commit/bbc4c3495b49b77468aeedeaee611be5c857a3f5

Should this part be using PackageDescription.EMPTY instead of PackageDescription.UNDEFINED?

@@ -1412,11 +1415,12 @@ public boolean isLocalType() {
         /**
          * {@inheritDoc}
          */
+        @Nullable
         public PackageDescription getPackage() {
             int packageIndex = name.lastIndexOf('.');
-            return new PackageDescription.Simple(packageIndex == -1
-                    ? EMPTY_NAME
-                    : name.substring(0, packageIndex));
+            return packageIndex == -1
+                    ? PackageDescription.UNDEFINED
+                    : new PackageDescription.Simple(name.substring(0, packageIndex));
         }

[raphw]

Yes, this is already fixed on master and I can confirm that it works with 1.12.14-SNAPSHOT. It's the mentioned inconsistency with handling the default package which should be resolved with the changes I made.

That said, the default package should be rather uncommon in actual Java applications, I hope this does not impact many users for it. I added some tests that use the default package to avoid further breakage.

[cushon]

Yes, this is already fixed on master

Ah, I hadn't seen https://github.com/raphw/byte-buddy/commit/aab1fe68c94cd28a1741a3dbb84c944747845863 yet, thanks.

I'll try upgrading again once 1.12.14 is available.