Could not deserialize session with CNFE Exception

[RemyaPR]

Hi @magro,

We have integrated with memcached back up session manager for HTTP session replication in Couchbase nodes.

Tomcat Context.xml Configuration

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="http://172.21.44.85:8091/pools"
    username="SessionBucket" operationTimeout="4000" sessionBackupAsync="false"  sessionBackupTimeout ="4000"  enabled="true"
    memcachedProtocol="binary" sessionAttributeFilter="^(?!.*fareCatalogHandlerMap).*$"
    requestUriIgnorePattern=".*(\.(ico|png|gif|jpg|css|js|js\?version=.*|css\?version=.*|properties\?.*|htm\?iconType=.*)|/ota/loadwtngpganim.htm\?isReloadPage=.*)$"
/>

At times we are seeing CNFE for a number of classes with trace as below

WARNING: Could not deserialize session with id 3E4CC1DFACCAA15EFBF4B906E4808088.node1 from memcached, session will be purged from storage.
de.javakaffee.web.msm.TranscoderDeserializationException: Caught CNFE decoding data
        at de.javakaffee.web.msm.JavaSerializationTranscoder.deserializeAttributes(JavaSerializationTranscoder.java:185)
        at de.javakaffee.web.msm.TranscoderService.deserializeAttributes(TranscoderService.java:170)
        at de.javakaffee.web.msm.TranscoderService.deserialize(TranscoderService.java:127)
        at de.javakaffee.web.msm.MemcachedSessionService.loadFromMemcached(MemcachedSessionService.java:1122)
        at de.javakaffee.web.msm.MemcachedSessionService.findSession(MemcachedSessionService.java:594)
        at de.javakaffee.web.msm.MemcachedBackupSessionManager.findSession(MemcachedBackupSessionManager.java:224)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2363)
        at org.apache.catalina.connector.Request.getSession(Request.java:2098)
        at org.apache.catalina.valves.AccessLogValve$SessionIdElement.addElement(AccessLogValve.java:1148)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:569)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.his.farebroker.search.vo.FareSearchVO
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603)
        at org.apache.catalina.util.CustomObjectInputStream.resolveClass(CustomObjectInputStream.java:78)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
        at de.javakaffee.web.msm.JavaSerializationTranscoder.deserializeAttributes(JavaSerializationTranscoder.java:172)
        ... 15 more 

This is not always but we are unable to identify a consistent behavior. Could you please help us to understand possible scenarios of getting this kind of error. We get this error even for spring framework class. Line 398: Caused by: java.lang.ClassNotFoundException: org.springframework.security.context.SecurityContextImpl

[magro]

Hi @RemyaPR, sounds weird, and classloaders are one of the weirdest things in java :-/ To be honest I have no clue what the reason might, asking some questions to achieve a better understanding.

• Which msm version are you using?
• tomcat version?
• jvm version?
• jvm settings/flags?
• Can you share which msm related libs are living where (which are in $CATALINA_HOME/lib/ and which are in WEB-INF/lib/)?
• Did this happen from day 1 you're using msm?
• Does this happen in production, during "standard" operation?
• Were there any "events" before this happens, like reload or anything else "unusual"?
• Does this happen (most often) with certain classes only?
• Can you reproduce/provoke this behaviour in a synthetic environment (on a dev machine)?
• Maybe you can try to use kryo and see if this resolves the issue (transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory")

[muten84]

Hi i'have same issue ,CNFE and SpringSecurityContext. I could check that all not found classes was object stored in session, and it happens randomly to not all sessions, (on 58 sessions, 3 or 4 of them was affected by this beahvior).

I'm using java jdk1.8.0_73 Tomcat 8.0.36 This is my memcahed sessione manager configuration

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
		memcachedNodes="n1:10.118.32.107:11211,n2:10.118.32.108:11211"
 		failoverNodes="n1" 
		requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$" 
	/>

Memcached version: VERSION 1.4.4 catalina home lib:

• memcached-session-manager-1.9.7
• memcached-session-manager-tc8-1.9.7
• spymemcached-2.7.3

It happens in production with same classes, i have thinked about different version of same class in classpath but if was that it should fail with all sessions. And it happened only one time in 5 years of production use.

i can't reproduce the problem.

why u suggest to use kryo ? should i use it?

[magro]

I asked to try kryo to see if it's related to the serialization itself or more a general classloader issue. Of course you could give it a try.