dubbo icon indicating copy to clipboard operation
dubbo copied to clipboard
verified publisher icon apache

[2.7.16] dubbo协议反序列化mongo-java-driver 4.2.0以上版本ObjectId失败

Open sunhk opened this issue 3 years ago • 9 comments

  • [x] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

  • Dubbo version: 2.7.16
  • Operating System version: windows 11
  • Java version: 1.8

Steps to reproduce this issue

在mongo-java-driver 4.2.0+版本中，ObjectId序列化方式发生变化 https://github.com/mongodb/mongo-java-driver/commit/57a20d675e0cdd1ae378b1b9ce969612953546a6 通过Hessian2Output写出到provider端时，Hessian2Input解析到的类型是org.bson.types.ObjectId.SerializationProxy 在调用com.alibaba.com.caucho.hessian.io.JavaDeserializer#readObject(com.alibaba.com.caucho.hessian.io.AbstractHessianInput, java.lang.String[])方法时，会试图利用反射构造其实例，调用构造方法org.bson.types.ObjectId.SerializationProxy#SerializationProxy时会报空指针

SerializationProxy(final ObjectId objectId) {
            bytes = objectId.toByteArray();
}

类似的的可定义类

   @ToString
    public class TimeRecord implements Serializable {
        private static final long serialVersionUID = 1L;

        private final String timestamp;

        public TimeRecord() {
            this(System.currentTimeMillis());
        }

        public TimeRecord(final long timestamp) {
            this.timestamp = String.valueOf(timestamp);
        }

        private long toLong() {
            return Long.parseLong(timestamp);
        }

        private Object writeReplace() {
            return new SerializationProxy(this);
        }

        private void readObject(final ObjectInputStream stream) throws InvalidObjectException {
            throw new InvalidObjectException("Proxy required");
        }

        public static class SerializationProxy implements Serializable {
            private static final long serialVersionUID = 1L;

            private final long number;

            SerializationProxy(final TimeRecord o) {
                number = o.toLong();
            }

            private Object readResolve() {
                return new TimeRecord(number);
            }
        }
    }

在consumer端任意service方法调用前附加隐式参数RpcContext.getClientAttachment().setObjectAttachment("haha", new TimeRecord());即可复现问题

Expected Behavior

希望能正确反序列化ObjectId，以便兼容更高版本的mongodb server； 或者是否可提供扩展接口，方便用户自定义特殊类型的bean的序列化方式 另外 Exception的message组织是否有问题，好多重复信息

Actual Behavior

exception trace:

org.apache.dubbo.rpc.RpcException: Failed to invoke the method sayHello in the service com.example.api.DemoService. Tried 3 times of the providers [192.168.199.1:30880] (1/1) from the registry 127.0.0.1:2181 on the consumer 192.168.199.1 using the dubbo version 3.0.6. Last error is: Failed to invoke remote method: sayHello, provider: dubbo://192.168.199.1:30880/com.example.api.DemoService?application=provider&background=false&category=providers,configurators,routers&check=false&deprecated=false&dubbo=2.0.2&interface=com.example.api.DemoService&metadata-type=remote&pid=24752&qos.enable=false&release=3.0.6&side=consumer&sticky=false, cause: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:317)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:202)
	at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:548)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2850)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2773)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2747)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.doReadMap(MapDeserializer.java:146)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.readMap(MapDeserializer.java:126)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2123)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2104)
	at org.apache.dubbo.common.serialize.hessian2.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:102)
	at org.apache.dubbo.common.serialize.ObjectInput.readAttachments(ObjectInput.java:87)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:224)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:93)
	at org.apache.dubbo.remoting.transport.DecodeHandler.decode(DecodeHandler.java:57)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:44)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:313)
	... 22 more
Caused by: java.lang.NullPointerException
	at com.example.Application$TimeRecord.access$000(Application.java:55)
	at com.example.Application$TimeRecord$SerializationProxy.<init>(Application.java:87)
	... 27 more

	at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:114)
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:340)
	at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:321)
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:99)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:321)
	at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:321)
	at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:108)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:321)
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:193)
	at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:92)
	at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:97)
	at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:280)
	at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:97)
	at com.example.api.DemoServiceDubboProxy0.sayHello(DemoServiceDubboProxy0.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	at com.sun.proxy.$Proxy56.sayHello(Unknown Source)
	at com.example.Application.main(Application.java:47)
Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:317)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:202)
	at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:548)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2850)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2773)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2747)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.doReadMap(MapDeserializer.java:146)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.readMap(MapDeserializer.java:126)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2123)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2104)
	at org.apache.dubbo.common.serialize.hessian2.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:102)
	at org.apache.dubbo.common.serialize.ObjectInput.readAttachments(ObjectInput.java:87)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:224)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:93)
	at org.apache.dubbo.remoting.transport.DecodeHandler.decode(DecodeHandler.java:57)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:44)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:313)
	... 22 more
Caused by: java.lang.NullPointerException
	at com.example.Application$TimeRecord.access$000(Application.java:55)
	at com.example.Application$TimeRecord$SerializationProxy.<init>(Application.java:87)
	... 27 more

	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:195)
	at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:264)
	at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:185)
	at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:378)
	at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:80)
	... 23 more
Caused by: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.example.Application$TimeRecord$SerializationProxy' could not be instantiated
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:317)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:202)
	at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:548)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2850)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2773)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2747)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2308)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.doReadMap(MapDeserializer.java:146)
	at com.alibaba.com.caucho.hessian.io.MapDeserializer.readMap(MapDeserializer.java:126)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2123)
	at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2104)
	at org.apache.dubbo.common.serialize.hessian2.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:102)
	at org.apache.dubbo.common.serialize.ObjectInput.readAttachments(ObjectInput.java:87)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:224)
	at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:93)
	at org.apache.dubbo.remoting.transport.DecodeHandler.decode(DecodeHandler.java:57)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:44)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:313)
	... 22 more
Caused by: java.lang.NullPointerException
	at com.example.Application$TimeRecord.access$000(Application.java:55)
	at com.example.Application$TimeRecord$SerializationProxy.<init>(Application.java:87)
	... 27 more

	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:214)
	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:176)
	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:164)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:60)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:181)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
	at org.apache.dubbo.common.threadpool.ThreadlessExecutor$RunnableWrapper.run(ThreadlessExecutor.java:184)
	at org.apache.dubbo.common.threadpool.ThreadlessExecutor.waitAndDrain(ThreadlessExecutor.java:103)
	at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:193)
	... 28 more

sunhk avatar Jul 29 '22 07:07 sunhk

SerializationProxy 这个类看起来就不太 Serializable 呀……

AlbumenJ avatar Aug 01 '22 06:08 AlbumenJ

image 然后这个方法和 hessian 预留的方法名刚好一起对上了

AlbumenJ avatar Aug 01 '22 06:08 AlbumenJ

是有点别扭，但官方文档https://docs.oracle.com/javase/8/docs/platform/serialization/spec/output.html#a5324 并未强调writeReplace的返回值需实现序列化接口，事实上SerializationProxy是否实现序列化接口也并不影响com.alibaba.com.caucho.hessian.io.JavaSerializer的逻辑 java原生的ObjectOutputStream与ObjectInputStream也可正常运行

@Test
    public void testObjectSerialization() throws IOException, ClassNotFoundException {
        // given
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(baos);
        ObjectId objectId = new ObjectId("5f8f4fcf27516f05e7eae5be");

        // when
        oos.writeObject(objectId);

        // then
        assertTrue(new String(baos.toByteArray()).contains("org.bson.types.ObjectId$SerializationProxy"));
        assertArrayEquals(new byte[] {-84, -19, 0, 5, 115, 114, 0, 42, 111, 114, 103, 46, 98, 115, 111, 110, 46, 116, 121, 112, 101, 115,
                        46, 79, 98, 106, 101, 99, 116, 73, 100, 36, 83, 101, 114, 105, 97, 108, 105, 122, 97, 116, 105, 111, 110, 80, 114,
                        111, 120, 121, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 1, 91, 0, 5, 98, 121, 116, 101, 115, 116, 0, 2, 91, 66, 120, 112, 117,
                        114, 0, 2, 91, 66, -84, -13, 23, -8, 6, 8, 84, -32, 2, 0, 0, 120, 112, 0, 0, 0, 12, 95, -113, 79, -49, 39, 81, 111,
                        5, -25, -22, -27, -66},
                baos.toByteArray());

        // when
        ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
        ObjectInputStream ois = new ObjectInputStream(bais);
        ObjectId deserializedObjectId = (ObjectId) ois.readObject();

        // then
        assertEquals(objectId, deserializedObjectId);
    }

sunhk avatar Aug 02 '22 02:08 sunhk

从逻辑上 com.alibaba.com.caucho.hessian.io.JavaSerializer 确实不应该和 SerializationProxy 有什么关系，现在的问题是 ObjectId 里面刚好有个方法名和 hessian 里面保留的方法名一样，导致走了特殊逻辑……

AlbumenJ avatar Aug 04 '22 03:08 AlbumenJ

也不能算“刚好”吧，writeReplace作为jdk提供给用户的自定义序列化方式之一，JavaSerializer试图提供支持（由#2031 可看出确实想支持此种序列化方式），虽然成功进行了序列化，但在反序列化过程中却没有恰当解析，强行调用有参构造方法而导致空指针。相对于对现有JavaDeserializer的完善，其实更想知道有没有关于com.alibaba.com.caucho.hessian.io.SerializerFactory的扩展，或者类似的接口，能对特定类型的序列化进行自定义

sunhk avatar Aug 04 '22 08:08 sunhk

额，好久没回复了，如果自定义扩展实现较复杂的话，是否可以改下JavaDeserializer中的对象实例化逻辑，借助Objenesis.jar来实例化对象

sunhk avatar Aug 12 '22 03:08 sunhk

看了下为啥 jdk 可以序列化，是因为 jdk 的直接建了个 constructor 出来，hessian 没有建的能力所以只能用 null 去填充

image image

AlbumenJ avatar Aug 12 '22 07:08 AlbumenJ

所以本质上 SerializationProxy 是无法按正常的方式序列化的，只是 jdk 底层给绕过了 constructor 的这个限制

AlbumenJ avatar Aug 12 '22 07:08 AlbumenJ

Objenesis应该也是判断jvm提供商来调用底层逻辑，虽然没细看，但感觉应该可以，反正代码是能跑通的

public class ReflectTest {
    public static void main(String[] args) throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException, IllegalAccessException, NoSuchFieldException {
        Objenesis objenesis = new ObjenesisStd();
        Class<?> clazz = Class.forName("org.bson.types.ObjectId$SerializationProxy",false,ReflectTest.class.getClassLoader());
        ObjectInstantiator<?> instantiator = objenesis.getInstantiatorOf(clazz);
        Object a = instantiator.newInstance();
        System.out.println(a);
        Field field = clazz.getDeclaredField("bytes");
        field.setAccessible(true);
        field.set(a,new byte[12]);
        Method method = clazz.getDeclaredMethod("readResolve");
        method.setAccessible(true);
        Object result = method.invoke(a);
        System.out.println(result);
    }
}

sunhk avatar Aug 15 '22 08:08 sunhk