apache
[ISSUE #9396] Use fastjson2 in all modules
Codecov Report
:x: Patch coverage is 58.06452% with 26 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 49.67%. Comparing base (1a3c922) to head (a7c4d8c).
Additional details and impacted files
@@ Coverage Diff @@
## develop #9397 +/- ##
=============================================
+ Coverage 48.39% 49.67% +1.28%
- Complexity 12286 12695 +409
=============================================
Files 1315 1315
Lines 93918 93931 +13
Branches 12046 12049 +3
=============================================
+ Hits 45452 46662 +1210
+ Misses 42854 41556 -1298
- Partials 5612 5713 +101
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
1、Could you provide the performance benchmark data for the original Fastjson 1? 2、Could you provide compatibility test results with Fastjson 1, particularly when the object contains camel-case properties?
1、Could you provide the performance benchmark data for the original Fastjson 1? 2、Could you provide compatibility test results with Fastjson 1, particularly when the object contains camel-case properties?
Thank you for your review. The data for the two questions are as follows:
1.
Regarding question 2, I encountered the camel case problem of getCId during the replacement test. The solution is to move the @JSONField(name = "c") annotation from the attribute to the get/set method. All the replaced codes are covered by unit tests.
Could you develop a tool to validate whether objects with camelCase properties are compatible with Fastjson 2, so we can check compatibility? @yx9o
Could you develop a tool to validate whether objects with camelCase properties are compatible with Fastjson 2, so we can check compatibility? @yx9o
Let me understand. We need to develop a tool to test all objects in the project that have irregular camel case naming to ensure that fastjson1 and fastjson2 serialization are compatible, right?
Could you develop a tool to validate whether objects with camelCase properties are compatible with Fastjson 2, so we can check compatibility? @yx9o
Let me understand. We need to develop a tool to test all objects in the project that have irregular camel case naming to ensure that fastjson1 and fastjson2 serialization are compatible, right?
Yes, but we should only validate classes involved in JSON serialization/deserialization.
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class: RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hello, I received your response. I have tested this class separately in the current branch and the develop branch. No errors occur. There is no "failed: xx" error output in the class itself. May I ask how you run this test?
@yx9o Subclasses of ConfigManager must also be checked
ConfigManager's subclasses are in multiple different packages and cannot be tested uniformly using tool classes. I want to do fastjson1 serialization and fastjson2 deserialization compatibility verification in each subclass's Test. Is this OK?
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hello, I received your response. I have tested this class separately in the current branch and the develop branch. No errors occur. There is no "failed: xx" error output in the class itself. May I ask how you run this test?
RemotingSerializableCompatTest.testCompatibilityCheck method , I change :
boolean isSuccess = checkCompatible(instance, clazz);
if (!isSuccess) {
System.out.println("failed:" + clazz);
}
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hello, I received your response. I have tested this class separately in the current branch and the develop branch. No errors occur. There is no "failed: xx" error output in the class itself. May I ask how you run this test?
RemotingSerializableCompatTest.testCompatibilityCheck method , I change :
boolean isSuccess = checkCompatible(instance, clazz); if (!isSuccess) { System.out.println("failed:" + clazz); }
I changed it like this and there was no error. Is there a difference in the environment? May I ask what branch, platform and jdk you are running on?
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hello, I received your response. I have tested this class separately in the current branch and the develop branch. No errors occur. There is no "failed: xx" error output in the class itself. May I ask how you run this test?
RemotingSerializableCompatTest.testCompatibilityCheck method , I change :
boolean isSuccess = checkCompatible(instance, clazz); if (!isSuccess) { System.out.println("failed:" + clazz); }I changed it like this and there was no error. Is there a difference in the environment? May I ask what branch, platform and jdk you are running on?
This PR branch and uses JDK 11.
There are many issues concerning compatibility reported at https://github.com/alibaba/fastjson2/issues.
Be extremely cautious Upgrading to fastjson2 involves high risks
Hi @fuyou001 , please help review again. According to your review, I added the following serialization compatibility test tool class:
RemotingSerializableCompatTest. It mainly does the following:
- Construct objects with default parameters for all subclasses of RemotingSerializable
- Use fastjson1 to serialize the object obtained in the first step
- Use fastjson2 to deserialize the json string obtained in the second step
- Compare the object deserialized in the third step with the object constructed in the first step, and compare all attribute values. Their comparisons are all passed
When running RemotingSerializableCompatTest, I encountered failed classes. Here's the output for your review: failed:class org.apache.rocketmq.remoting.protocol.body.RegisterBrokerBody failed:class org.apache.rocketmq.remoting.protocol.header.GetConsumerListByGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.route.TopicRouteData failed:class org.apache.rocketmq.remoting.protocol.statictopic.TopicRemappingDetailWrapper failed:class org.apache.rocketmq.remoting.protocol.body.SubscriptionGroupList failed:class org.apache.rocketmq.remoting.protocol.body.GroupList failed:class org.apache.rocketmq.remoting.protocol.body.ElectMasterResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.QueryAssignmentResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.BrokerMemberGroup failed:class org.apache.rocketmq.remoting.protocol.heartbeat.HeartbeatData failed:class org.apache.rocketmq.remoting.protocol.body.CheckClientRequestBody failed:class org.apache.rocketmq.remoting.protocol.body.GetBrokerMemberGroupResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ClusterInfo failed:class org.apache.rocketmq.remoting.protocol.body.QueryConsumeQueueResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.TopicList failed:class org.apache.rocketmq.remoting.protocol.body.QuerySubscriptionResponseBody failed:class org.apache.rocketmq.remoting.protocol.body.ConsumeByWho failed:class org.apache.rocketmq.remoting.protocol.body.ConsumerRunningInfo failed:class org.apache.rocketmq.remoting.protocol.body.KVTable failed:class org.apache.rocketmq.remoting.protocol.body.CreateTopicListRequestBody
Hello, I received your response. I have tested this class separately in the current branch and the develop branch. No errors occur. There is no "failed: xx" error output in the class itself. May I ask how you run this test?
RemotingSerializableCompatTest.testCompatibilityCheck method , I change :
boolean isSuccess = checkCompatible(instance, clazz); if (!isSuccess) { System.out.println("failed:" + clazz); }I changed it like this and there was no error. Is there a difference in the environment? May I ask what branch, platform and jdk you are running on?
This PR branch and uses JDK 11.
There are many issues concerning compatibility reported at https://github.com/alibaba/fastjson2/issues.
Be extremely cautious Upgrading to fastjson2 involves high risks
I have passed the debugging under jdk11, please help review it again, thank you.
@yx9o The conflicts need to be handled
The conflict has been resolved, please review again, thank you.
All fields in classes used for Fastjson serialization/deserialization must be annotated with @JSONField, requiring verification tools for annotation compliance.
@yx9o
@yx9o Some problems were encountered during testing
Corrected, please review and verify again, thank you.
@yx9o Some problems were encountered during testing yet
com.alibaba.fastjson2.JSONException: not support input EA==, offset 22, character ", line 1, column 23, fastjson-version 2.0.43 {"acks":[{"b":"EA==","c":"gid-72","it":60000,"pt":1758617828232,"q":0,"r":"0","rq":4,"so":245397,"t":"topic-a72"}],"brokerName":"rocketmq-broker-rmq-cn-htq4fg13h09-0"} at com.alibaba.fastjson2.JSONReader.readBinary(JSONReader.java:756) at com.alibaba.fastjson2.reader.ObjectReaderImplBitSet.readObject(ObjectReaderImplBitSet.java:44) at com.alibaba.fastjson2.reader.ObjectReaderImplBitSet.readObject(ObjectReaderImplBitSet.java:11) at com.alibaba.fastjson2.reader.ORG_20_18_BatchAck.readObject(Unknown Source) at com.alibaba.fastjson2.reader.ORG_18_2_BatchAckMessageRequestBody.readObject(Unknown Source) at com.alibaba.fastjson2.JSON.parseObject(JSON.java:1365) at org.apache.rocketmq.remoting.protocol.RemotingSerializable.fromJson(RemotingSerializable.java:64) at org.apache.rocketmq.remoting.protocol.RemotingSerializable.decode(RemotingSerializable.java:48) at org.apache.rocketmq.broker.processor.AckMessageProcessor.processRequest(AckMessageProcessor.java:170) at org.apache.rocketmq.broker.processor.AckMessageProcessor.processRequest(AckMessageProcessor.java:114) at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.lambda$buildProcessRequestHandler$2(NettyRemotingAbstract.java:426) at org.apache.rocketmq.remoting.netty.RequestTask.run(RequestTask.java:80) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:991)
@yx9o Some problems were encountered during testing yet
com.alibaba.fastjson2.JSONException: not support input EA==, offset 22, character ", line 1, column 23, fastjson-version 2.0.43 {"acks":[{"b":"EA==","c":"gid-72","it":60000,"pt":1758617828232,"q":0,"r":"0","rq":4,"so":245397,"t":"topic-a72"}],"brokerName":"rocketmq-broker-rmq-cn-htq4fg13h09-0"} at com.alibaba.fastjson2.JSONReader.readBinary(JSONReader.java:756) at com.alibaba.fastjson2.reader.ObjectReaderImplBitSet.readObject(ObjectReaderImplBitSet.java:44) at com.alibaba.fastjson2.reader.ObjectReaderImplBitSet.readObject(ObjectReaderImplBitSet.java:11) at com.alibaba.fastjson2.reader.ORG_20_18_BatchAck.readObject(Unknown Source) at com.alibaba.fastjson2.reader.ORG_18_2_BatchAckMessageRequestBody.readObject(Unknown Source) at com.alibaba.fastjson2.JSON.parseObject(JSON.java:1365) at org.apache.rocketmq.remoting.protocol.RemotingSerializable.fromJson(RemotingSerializable.java:64) at org.apache.rocketmq.remoting.protocol.RemotingSerializable.decode(RemotingSerializable.java:48) at org.apache.rocketmq.broker.processor.AckMessageProcessor.processRequest(AckMessageProcessor.java:170) at org.apache.rocketmq.broker.processor.AckMessageProcessor.processRequest(AckMessageProcessor.java:114) at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.lambda$buildProcessRequestHandler$2(NettyRemotingAbstract.java:426) at org.apache.rocketmq.remoting.netty.RequestTask.run(RequestTask.java:80) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:991)
The adjustment has been completed, please review again, thank you.
希望增加bitset的兼容性测试，在目前的测试中该字段是null，可以增加模拟值去比较兼容性。
Added, please review again, thank you.