EsotericSoftware
Encountered unregistered class ID: 50
Describe the bug
Got a exception: Encountered unregistered class ID: 50
Here is my code snippet
Here is the idToRegistration which in the kryo object.
There is no class with id equals to 50. How to find the unregistered class ?
To Reproduce
Environment:
- JDK Version: 21
- Kryo Version: 5.6.2
This can usually only happen if your data model or your Kryo config changes between serialization and deserialization. Can you reproduce this when you serialize and deserialize in a unit test?
This can usually only happen if your data model or your Kryo config changes between serialization and deserialization. Can you reproduce this when you serialize and deserialize in a unit test?
Is there a way to find all the classes that need to be registered ？
Can you explain in more detail what you mean by that or what you want to achieve?
Can you explain in more detail what you mean by that or what you want to achieve?
I have a program that processes a batch of data every minute and then serializes the object and writes it to rocksdb. In the next minute, the data is got from the rocksdb and deserialized into an object for processing. We chose kryo for serialization and deserialization because it is excellent in terms of performance and size. But in the process of actual operation, can appear sometimes "Encountered unregistered class ID: 50" and "KryoBufferUnderflowException: Buffer underflow" exception. Since our data is relatively sensitive, I am trying to make a reproducible data.
Understood.
As I said before, this error usually only happens if deserialization happens with a different Kryo configuration or if your classes changed between serialization and deserialization. Are you sure that your configuration and your classes are exactly the same for deserialization? Does deserialization happen on the same machine that serialized the data or is this a distributed system?
Since our data is relatively sensitive, I am trying to make a reproducible data.
Please do!
Understood.
As I said before, this error usually only happens if deserialization happens with a different Kryo configuration or if your classes changed between serialization and deserialization. Are you sure that your configuration and your classes are exactly the same for deserialization? Does deserialization happen on the same machine that serialized the data or is this a distributed system?
Since our data is relatively sensitive, I am trying to make a reproducible data.
Please do!
Because it is read and write rocksdb, it is all done on one instance.
Understood.
As I said before, this error usually only happens if deserialization happens with a different Kryo configuration or if your classes changed between serialization and deserialization. Are you sure that your configuration and your classes are exactly the same for deserialization? Does deserialization happen on the same machine that serialized the data or is this a distributed system?
Since our data is relatively sensitive, I am trying to make a reproducible data.
Please do!
Hi theigl The attached is the code and data required for reproduction.
Not yet.
Is it strictly necessary for your reproducer to work on binary data from rocksdb? Or can you reproduce this in a real unit test. I.e. serialize the data to a byte[] and deserialize it right away in a single test?
Because if you can't reproduce it in a simple unit test, then the problem must be with how you store or read the data from rocksdb. The error you are seeing usually only happens for one of these reasons:
- Incorrect usage of Kryo (multiple threads using the same instance, wrong usage of
Inputetc) - Kryo config changed between serialization and deserialization
- Data model changed between serialization and deserialization
- Data got corrupted during storage or reading from storage
Can you try to adjust your test to not require a roundtrip to rocksdb?
Hi, @theigl !
We recently encountered similar issue. And we are unable to reproduce it in a local environment or with a unittest.
We have the issue only on a cloud k8s cluster environment. When we load the same data locally we dont have the issue.
We write the data to Redis in a byte array format.
When we read the data we get the unregistered class ID error on some items.
To register our classes we do:
ThreadLocal<Kryo> kryoThreadLocal = ThreadLocal.withInitial(() -> { Kryo kryo new Kryo(); ClassRegistryHelper.registerClasses(kryo); return kryo; });
We then get a Kryo instance from kryoThreadLocal each time we serialize or deserialize data.
Is this approach right?
@ivan4oto: Yeah that usage looks alright.
We have the issue only on a cloud k8s cluster environment. When we load the same data locally we dont have the issue. We write the data to Redis in a byte array format. When we read the data we get the unregistered class ID error on some items.
Are you able to reproduce this somehow or is it happening only once a while in production? I'm guessing that your data gets modified/corrupted in some way when it is stored or read from Redis. It would be interesting to see a diff between the data that gets read from Redis and the correct byte data.
I'm also using Kryo to store data in Redis and never had an issue like that.
One thing that both of you could try is this:
kryo.setOptimizedGenerics(false);
This disabled generics optimization. It will slightly increase the serialized size of your data, but it disables the part of Kryo that was responsible for some rare bugs in the past. Beware that this setting is not backwards compatible. If you change it, you will have to re-serialize your data.
If that doesn't help, you can only try to get a diff between the correct data and the data from Redis and compare the two.