Support for JDK17

[dkroehan]

Describe the bug It seems that Kryo is currently not compatible with JDK17. When calling the default constructor it fails with:

java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
	at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
	at com.esotericsoftware.kryo.unsafe.UnsafeUtil.<clinit>(UnsafeUtil.java:100)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at com.esotericsoftware.kryo.util.Util.<clinit>(Util.java:47)
	at com.esotericsoftware.kryo.serializers.OptionalSerializers.addDefaultSerializers(OptionalSerializers.java:38)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:230)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:168)

To Reproduce Kryo kryo = new Kryo();

Environment:

• JDK Version: OpenJDK 17.0.2 (Corretto-17.0.2.8.1)
• Kryo Version: 5.3.0

Additional context In my case I am also registering some standard classes. It seems to work for classes like:

• java.util.Locale
• java.time.Instant
• java.util.HashMap

But it fails for java.util.regex.Pattern due to:

java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.util.regex.Pattern.pattern accessible: module java.base does not "opens java.util.regex" to unnamed module @185a6e9

What would be the proper way to fix this?

[theigl]

Your specific case can be solved by using a custom RegexSerializer that doesn't rely on reflection.

Alternatively, you can add JVM arguments for opening up the modules in question. See this comment.

I think it makes sense to include the RegexSerializer and UUIDSerializer directly in Kryo, so those two classes can be serialized on JDK 17 without opening any modules.

[theigl]

I cannot reproduce your first issue. I just downloaded and ran my tests on Corretto to make sure it isn't a JVM specific problem.

[dkroehan]

Thank you for the quick response.

I did set a breakpoint at AccessibleObject#checkCanSetAccessible at line 354 where an InaccessibleObjectException is created for debugging purposes. This is where I found: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d

This is caused by the Util static initializer block at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/util/Util.java#L47 This calls UnsafeUtil with the static initializer at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/unsafe/UnsafeUtil.java#L99

The InaccessibleObjectException is caught in line 101, which causes the directByteBufferConstructor to be null. I didn't pay attention that the Exception is actually not bubbling up.

Now I'm wondering if this is expected behaviour or a failure with java 17.

[dkroehan]

I tried the RegexSerializer using:

kryo.register(Pattern.class, new RegexSerializer());

It seems to work.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated: #https://github.com/magro/kryo-serializers/issues/131#issuecomment-1046577690

[theigl]

Now I'm wondering if this is expected behaviour or a failure with java 17.

This is expected behavior and has no effect unless you want to use UnsafeByteBuffers for Input/Output. If you want to use those, you have to open the module up for reflection.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated:

AFAIK it is impossible to write a non-reflective version of a serializer for UnmodifiableCollection that covers all potential cases. So these serializers will probably never make it into Kryo and you will have to add add-opens arguments if you need to serialize them.

[theigl]

Kryo is currently missing built-in, safe serializers for the following JDK classes:

• java.net.URI
• java.util.UUID
• java.util.regex.Pattern
• java.util.concurrent.atomic.AtomicBoolean
• java.util.concurrent.atomic.AtomicInteger
• java.util.concurrent.atomic.AtomicLong
• java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

[jdorleans]

I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
	at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

[theigl]

@jdorleans: On JDK 17, you currently need to open up sun.nio.ch like this:

--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

Otherwise initialization of UnsafeUtil fails because it cannot run DirectBuffer.class.getMethod("cleaner"). We should allow usage of unsafe without requiring access to DirectBuffer. I'll investigate how other libraries solve this. For now, please add the --add-opens statement.

[graves501]

I guess I'm just doing something wrong, but I'm also currently trying the latest kryo version (5.3.0) in combination with Java 17. I'm trying to clone an object that contains a list.

final var kryo = new Kryo();
kryo.register(ClassWithList.class);
ClassWithList copy = kryo.copy(classWithList);

I get this error messsage:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: java.util.ImmutableCollections$List12
Note: To register this class use: kryo.register(java.util.ImmutableCollections.List12.class);

Registering this class doesn't work, since List12 is not publicly accessible.

[theigl]

@graves501: You can either set registrationRequired(false) or manually register the classes using one of the following:

kryo.register(List.of(1).getClass());
kryo.register(List.of(1,2,3).getClass());

kryo.register(Class.forName("java.util.ImmutableCollections$List12"));
kryo.register(Class.forName("java.util.ImmutableCollections$ListN"));

[graves501]

Thanks, that did the job!

[theigl]

Quick update:

1. I added the following serializers to Kryo. These serializers are not registered by default (because of backwards compatibility), but can be registered as needed:

• java.net.URI
• java.util.UUID
• java.util.regex.Pattern
• java.util.concurrent.atomic.AtomicBoolean
• java.util.concurrent.atomic.AtomicInteger
• java.util.concurrent.atomic.AtomicLong
• java.util.concurrent.atomic.AtomicReference
  See https://github.com/EsotericSoftware/kryo/pull/930

2. It should no longer be necessary to open up sun.nio.ch. Access to DirectBuffer is delayed until it is first used.
   See https://github.com/EsotericSoftware/kryo/pull/932

3. I added a helper method to register serializers for ImmutableCollections:
   ImmutableCollectionsSerializers.registerSerializers(kryo);
   See https://github.com/EsotericSoftware/kryo/pull/933

It would be great if anyone could confirm the second change using the current snapshots.

[leonchen83]

@theigl

Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make field private transient java.lang.Object java.lang.Throwable.backtrace accessible: module java.base does not "opens java.lang" to unnamed module @2eafffde
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.CachedFields.addField(CachedFields.java:123)
	at com.esotericsoftware.kryo.serializers.CachedFields.rebuild(CachedFields.java:99)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:82)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:124)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:108)
	at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:469)
	at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:454)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:89)
	at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:581)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:112)
	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:708)
	at cn.nextop.test.Main.writeClassAndObject(Main.java:59)
	at cn.nextop.test.Main.main(Main.java:37)

version: kryo 5.4.0

reproduce code

import java.util.concurrent.ExecutionException;

import org.objenesis.strategy.StdInstantiatorStrategy;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.unsafe.UnsafeInput;
import com.esotericsoftware.kryo.unsafe.UnsafeOutput;
import com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy;

public class Main {
	public static void main(String[] args) {
		Main main = new Main();
		
		Kryo kryo = main.init();
		ExecutionException rpc = new ExecutionException(null, new Throwable());
		byte[] bytes = main.writeClassAndObject(kryo, rpc);
		ExecutionException value = (ExecutionException) main.readClassAndObject(kryo, bytes);
	}
	
	public Kryo init() {
		Kryo kryo = new Kryo();
		kryo.setReferences(true);
		kryo.setRegistrationRequired(false);
		kryo.setWarnUnregisteredClasses(false);
		
		DefaultInstantiatorStrategy init = (DefaultInstantiatorStrategy) kryo.getInstantiatorStrategy();
		init.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
		return kryo;
	}
	
	public Object readClassAndObject(Kryo kryo, byte[] bytes ) {
		UnsafeInput input = new UnsafeInput();
		input.setBuffer(bytes); Object r = kryo.readClassAndObject(input); return r;
	}
	
	public byte[] writeClassAndObject (Kryo kryo, Object value) {
		UnsafeOutput output = new UnsafeOutput(1024, 32 * 1024 * 1024);
		kryo.writeClassAndObject(output, value); return output.toBytes();
	}
}

We need to serialize ExecutionException because we need to implement an RPC service by kryo. so could we add ExecutionExceptionSerializer to kryo 5.4.0?

[theigl]

@leonchen83: I would not consider serializing exceptions a typical use-case for Kryo, so I'm against adding exception serializers to Kryo itself.

You have two options:

1. Write your own serializer
2. Use JavaSerializer for serializing exceptions

I'd go for the second option. Exceptions should occur very rarely, so speed of serialization and payload size usually doesn't matter that much.

Redisson for instance is using this approach.

[amodolo]

Hi @theigl, also ClosureSerializer is not compatible with JDK 17. Is there any workarround to solve this

Version 5.5.0 JDK: temurin 17.0.7

Here is the code used to reproduce the exception

public static void main(String[] args) {
    Kryo kryo = new Kryo();
    kryo.register(ClosureSerializer.Closure.class, new ClosureSerializer());
}

here is the exception

00:00  WARN: Unable to obtain SerializedLambda#readResolve via reflection. Falling back on resolving lambdas via capturing class.
java.lang.reflect.InaccessibleObjectException: Unable to make private java.lang.Object java.lang.invoke.SerializedLambda.readResolve() throws java.io.ObjectStreamException accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:59)
	at dev.test.serialization.kryo.Main.main(Main.java:9)
00:00  WARN: Unable to obtain SerializedLambda#capturingClass via reflection. Falling back to resolving capturing class via Class.forName.
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.lang.Class java.lang.invoke.SerializedLambda.capturingClass accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:69)
	at dev.test.serialization.kryo.Main.main(Main.java:9)

I've already try to set this argument to the JVM

--add-opens=java.base/java.lang.invoke=ALL-UNNAMED 

but it solves nothing

[theigl]

@amodolo: I am using ClosureSerializer on OpenJDK 17 for quite a while now. It works fine. Are you sure you are passing the JVM argument correctly? I'm passing the following argument:

--add-opens java.base/java.lang.invoke=ALL-UNNAMED

Maybe the additional = causes problems in your case?

[amodolo]

You are right. there is an additional =. even without it, the problem is still there. Here is my configuration.

[theigl]

I just ran ClosureSerializerTest in Kryo with the following settings:

It works fine.

The command run by IntelliJ looks like this:

/Users/XXX/Library/Java/JavaVirtualMachines/temurin-17.0.7/Contents/Home/bin/java --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --enable-preview ......

How does it look in your case?

[amodolo]

Oh i see what's going wrong. i passed the --add-opens to the program arguments instead of VM options. Your configuration solves the problem. Thank you

[endlesstian]

Kryo is currently missing built-in, safe serializers for the following JDK classes:

• java.net.URI
• java.util.UUID
• java.util.regex.Pattern
• java.util.concurrent.atomic.AtomicBoolean
• java.util.concurrent.atomic.AtomicInteger
• java.util.concurrent.atomic.AtomicLong
• java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

Hi, when will Kryo 6 release? Thanks. BRs, Tian

[theigl]

@endlesstian: You can already register all these serializers yourself. They are available for manual registration since Kryo 5.4.0.

As for Kryo 6, I currently do not have a release date. I will have more spare time in December and I'll see if I can create a release candidate by the end of the year.

[vyalyh-oleg]

Can we expect, that the rkyo library will support modular system ? --add-opens isn't a problem, but I would like to open the specific package not to all UNNAMED modules, but e.g. to a specific one (kryo). I think it could solve some problems.

[theigl]

@vyalyh-oleg: Kryo currently only adds an automatic module name. So far, there is no module definition. What is required for add-opens to be restricted to Kryo?