eclipselink
eclipselink copied to clipboard
eclipse-ee4j
Configuration of CACHE_QUERY_FORCE_DEFERRED_LOCKS in org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader) taking place too late
Please be aware,
We currently have a tracker open in oracle related to dead locks in production due to a bug in the: org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.updateDescriptorCacheSettings(Map, ClassLoader)
In eclipse link 2.7.6 there was the problem that the line of code:
String queryCacheForceDeferredLocks = getConfigPropertyAsStringLogDebug(PersistenceUnitProperties.CACHE_QUERY_FORCE_DEFERRED_LOCKS, m, session);
session.getProject().setQueryCacheForceDeferredLocks("true".equalsIgnoreCase(queryCacheForceDeferredLocks));
Was missing. But for this bug there are official oracle patches.
But then all subsequent version, including the https://github.com/eclipse-ee4j/eclipselink/blob/2.7.13/jpa/org.eclipse.persistence.jpa/src/org/eclipse/persistence/internal/jpa/EntityManagerSetupImpl.java
There is the problem that the line of code is placed too late.
The line code is being placed after:
Map typeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_TYPE_, m, session);
Map sizeMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SIZE_, m, session);
Map sharedMap = PropertiesHandler.getPrefixValuesLogDebug(PersistenceUnitProperties.CACHE_SHARED_, m, session);
if(typeMap.isEmpty() && sizeMap.isEmpty() && sharedMap.isEmpty()) {
return;
}
And this a gigantic problem because as it turns out, the weblogic deloyments seem to be coming out on that line code.
Our persistence.xml had the following configuration properties to mitigate the risk of dead locks:
<!-- Object Building MUST ALWAYS use deferred locks. If cannot acquire it must let it go and continue forward -->
<!-- waiting for whoever acquired the lock to arrive at the releaseDeferredLocks moment as well. -->
<!-- The threads committing changes they automatically let go of the write locks when they fail to acquire them all -->
<property name="eclipselink.cache.query-force-deferred-locks" value="true"/>
<!-- Other properties that we can use -->
<!-- The properties we typically configure as system properties: -->
<!-- org.eclipse.persistence.config.SystemProperties.CONCURRENCY_MANAGER_ACQUIRE_WAIT_TIME -->
<!-- For example set JAVA_OPTIONS=%JAVA_OPTIONS% -Declipselink.concurrency.manager.waittime=10 -->
<!-- They typically are also specified as pseristence unit properties: -->
<!-- org.eclipse.persistence.config.PersistenceUnitProperties.CONCURRENCY_MANAGER_ACQUIRE_WAIT_TIME -->
<!-- Persistence.xml properties that we typically configure with system property. -->
<property name="eclipselink.concurrency.manager.waittime" value="10"/>
<property name="eclipselink.concurrency.manager.maxsleeptime" value="10000"/>
<property name="eclipselink.concurrency.manager.maxfrequencytodumptinymessage" value="5000"/>
<property name="eclipselink.concurrency.manager.maxfrequencytodumpmassivemessage" value="20000"/>
<property name="eclipselink.concurrency.manager.allow.interruptedexception" value="false"/>
<property name="eclipselink.concurrency.manager.allow.exception" value="false"/>
<property name="eclipselink.concurrency.manager.allow.concurrency.exception" value="false"/>
<property name="eclipselink.concurrency.manager.allow.readlockstacktrace" value="false"/>
<property name="eclipselink.concurrency.manager.object.building.semaphore" value="true"/>
<property name="eclipselink.concurrency.manager.write.lock.manager.semaphore" value="true"/>
<property name="eclipselink.concurrency.manager.object.building.no.threads" value="12"/>
<property name="eclipselink.concurrency.manager.write.lock.manager.no.threads" value="2"/>
<property name="eclipselink.concurrency.semaphore.max.time.permit" value="2000"/>
<property name="eclipselink.concurrency.semaphore.log.timeout" value="2000"/>
But the holy grail of the config properties:
Was not doing anything.
So we had the typical dead locks with some threads waiting to release deferred locks. Those are the nice threads that be having properly.
```
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:653)
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1107)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInternal(ObjectBuilder.java:774)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:727)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:705)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:861)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:570)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:447)
at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:3367)
So this part was looking good in the dead locks. These are the threads we know that cause no trouble.
But then we had the nasty ones;
"[STUCK] ExecuteThread: '69' for queue: 'weblogic.kernel.Default (self-tuning)'" #366 daemon prio=1 os_prio=0 cpu=11193.21ms elapsed=1815.58s tid=0x00007ff6e405f000 nid=0x37686 in Object.wait() [0x00007ff395595000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:128)
- waiting to re-lock in wait() <0x0000000309d45180> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:139)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:110)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:175)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:101)
at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.acquireLock(IsolatedClientSessionIdentityMapAccessor.java:78)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:92) <------- Tthis is a complete application distructor code line, the retrieveCacheKey should be calling for the
----- cacheKey = this.getIdentityMapAccessorInstance().acquireDeferredLock(primaryKey, concreteDescriptor.getJavaClass(), concreteDescriptor, query.isCacheCheckComplete() || query.shouldRetrieveBypassCache());
at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5383) <-----Was choosing the wrong code path
So please be aware of this defect.
I would also very strongly recommend, please make the eclipselink default have the force deferred locks active to TRUE and put in capital letters a strong recommendation against ever setting it to false.
The constructor of the:
org.eclipse.persistence.queries.ObjectBuildingQuery.ObjectBuildingQuery()
Should probably default the
` this.requiresDeferredLocks = true;`
and finally the
org.eclipse.persistence.queries.ObjectBuildingQuery.requiresDeferredLocks()
In case of null should return true.
Thanks for considering these suggestions
