Error on reconnecting to distributed database

[knuthelgesen]

OrientDB Version: 3.1.2, running in Docker based on official Docker image

Java Version: 1.8.0_265

OS: Docker host is SLES-12

Expected behavior

The clients can reconnect to a distributed database with a connection string specifying all the hostsnames.

Actual behavior

We have three Docker hosts, and on each one there is a container running OrientDB. These are all part of the same cluster, which works fine. The client connects to the servers using the code: new OrientGraphFactory("remote:docker01;docker02;docker03/portal "user", "pw").setupPool(1, 50) This also works fine.

However, after a while of inactivity, the connection from the client to the server goes stale or times out. And the client is not able to reconnect to the server.

We see the following error message in the client log: 2020-09-16 18:18:43.169 INFO 16 --- [nio-8080-exec-2] c.o.orient.client.remote.OStorageRemote : Caught Network I/O errors on docker03:2424/portal, trying an automatic reconnection... (error: null) 2020-09-16 18:18:43.209 ERROR 16 --- [nio-8080-exec-2] n.k.b.p.rest.controller.BildeController : Feilmelding: null com.orientechnologies.common.io.OIOException: null at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:497) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:365) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:405) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1095) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:72) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:76) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:84) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:34) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.query(ODatabaseDocumentAbstract.java:358) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:769) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.tinkerpop.blueprints.impls.orient.OrientGraphQuery.vertices(OrientGraphQuery.java:206) ~[orientdb-graphdb-3.1.2.jar!/:3.1.2] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertices(OrientBaseGraph.java:860) ~[orientdb-graphdb-3.1.2.jar!/:3.1.2] at no.kommune.bergen.plattform.transaksjon.TransactionWrapper.findByProperty(TransactionWrapper.java:130) ~[domeneplattform-12.0-SNAPSHOT.jar!/:na] at no.kommune.bergen.plattform.transaksjon.TransactionWrapper.findById(TransactionWrapper.java:138) ~[domeneplattform-12.0-SNAPSHOT.jar!/:na] at no.kommune.bergen.portal.rest.controller.BildeController.get(BildeController.java:53) ~[classes!/:na] at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at no.kommune.bergen.portal.filter.SiteFilter.doFilter(SiteFilter.java:47) [portal-domene-12.0-SNAPSHOT.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at no.kommune.bergen.portal.filter.TransactionFilter.doFilter(TransactionFilter.java:33) [portal-domene-12.0-SNAPSHOT.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.7.RELEASE.jar!/:5.1.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:836) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1747) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.19.jar!/:9.0.19] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: java.io.EOFException: null at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[na:1.8.0_212] at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.readByte(OChannelBinary.java:78) ~[orientdb-core-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:185) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:169) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2221) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:385) ~[orientdb-client-3.1.2.jar!/:3.1.2] at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:447) ~[orientdb-client-3.1.2.jar!/:3.1.2] ... 73 common frames omitted

And the following error in the server log: Error on unmarshalling content. Class: q com.orientechnologies.orient.core.exception.ODatabaseException: The database instance is not set in the current thread. Be sure to set it with: ODatabaseRecordThreadLocal.instance().set(db); at com.orientechnologies.orient.core.db.ODatabaseRecordThreadLocal.get(ODatabaseRecordThreadLocal.java:61) at com.orientechnologies.orient.core.sql.query.OSQLQuery.deserializeQueryParameters(OSQLQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.queryFromStream(OSQLQuery.java:137) at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.queryFromStream(OSQLSynchQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:109) at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:86) at com.orientechnologies.orient.client.remote.message.OCommandRequest.read(OCommandRequest.java:70) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:319) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67) Error reading request com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling content. Class: q at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:92) at com.orientechnologies.orient.client.remote.message.OCommandRequest.read(OCommandRequest.java:70) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:319) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67) Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: The database instance is not set in the current thread. Be sure to set it with: ODatabaseRecordThreadLocal.instance().set(db); at com.orientechnologies.orient.core.db.ODatabaseRecordThreadLocal.get(ODatabaseRecordThreadLocal.java:61) at com.orientechnologies.orient.core.sql.query.OSQLQuery.deserializeQueryParameters(OSQLQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.queryFromStream(OSQLQuery.java:137) at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.queryFromStream(OSQLSynchQuery.java:147) at com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:109) at com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:86) ... 4 more

Steps to reproduce

1. Set up a distributed database with three nodes.
2. Connect using all the hostsnames in the connectionstring.
3. Wait for the connection to time out.
4. Attempt reconnection.

[lvca]

This issue should be fixed in the most recent version. Please use the latest 3.1.x.