Database connection leak using JPA and loading lazy initialized entities using WebFlow's presistence context [SWF-1525]

[spring-operator]

Artem Karpenko opened SWF-1525 and commented

When entities have lazy-initialized relations and they are loaded using WebFlow's persistence context (without transaction) then database connection is not closed but at the same time is not reused later - i.e. it's just lost. This problem initially appeared in working project and I was able to reproduce it in small test project (attached).

About sample. This project is based on booking-faces example. The main changes were done in webflow-config.xml (added jpaFlowExecutionListener), data-access-config.xml (using MySQL and tomcat-jdbc connection pool (original application with this problem used DBCP pool)) and main-flow.xml (greatly simplified). Also a new entity, Suite, was added as one-to-many relation to Hotel in order to add lazy initialization. On start of the main flow a Hotel instance is loaded using transaction method from service and then its lazy relation is initialized using WebFlow's entity manager.

How to reproduce problem. Deploy and start application (also start mysql server with booking_faces database). On start page click "Start your Spring Travel experience". That's it, one connection is leaked (this was checked using a heap dump). Now click on top Spring logo at the top to return to main page. Do the same two more times. Now when you'll click on start link for the forth time (maxActive connections for the pool is set to 3) an exception will pop up:

 
java.sql.SQLException: [http-8080-2] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:3; busy:3; idle:0; lastwait:30000].
	org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:638)
	org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:179)
	org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:124)
	org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
	org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
	org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
	org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
	org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
	org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1463)
	org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60)
	org.springframework.orm.jpa.DefaultJpaDialect.beginTransaction(DefaultJpaDialect.java:70)
	org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:57)
	org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:332)
	org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
	org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
	org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	$Proxy30.findHotelById(Unknown Source)
	localhost.MyForm.loadHotel(MyForm.java:16)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:597)
	org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69)
	org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:83)
	org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:57)
	org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102)
	org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:97)
	org.springframework.binding.expression.spel.SpringELExpression.getValue(SpringELExpression.java:84)
	org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:75)
	org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)
	org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145)
	org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)
	org.springframework.webflow.engine.ActionList.execute(ActionList.java:155)
	org.springframework.webflow.engine.Flow.start(Flow.java:534)
	org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:366)
	org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:222)
	org.springframework.webflow.executor.FlowExecutorImpl.launchExecution(FlowExecutorImpl.java:140)
	org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:193)
	org.springframework.faces.webflow.JsfFlowHandlerAdapter.handle(JsfFlowHandlerAdapter.java:48)
	org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
	org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
	org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:343)
	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:177)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

This issue is basically the same as #421 (was not sure whether I should just comment there?). Rossen Stoyanchev commented: ??JPA does not provide a configurable connection release mode and providers should always eagerly release JDBC connections??. Indeed, setting hibernate.connection.release_mode to after_statement resolves described problem. Is this a supposed solution? If it is then it should probably be documented, because Hibernate's documentation suggests using after_transaction for non-JTA environments.

---

Affects: 2.3.1

Attachments:

• booking-faces.zip (251.05 kB)

7 votes, 8 watchers

[spring-operator]

Artem Karpenko commented

Having read about your GitHub-based system for demo projects just now I've added one: no JSF, based on SWF-0000.

[spring-operator]

Rossen Stoyanchev commented

Thanks for providing the sample code. I've modified the project slightly to use HSQLDB.

[spring-operator]

Fabricio Colombo commented

We have the same issue and this appear does not be a spring bug, but a hibernate behaviour how related at https://hibernate.onjira.com/browse/HHH-4808

Using spring webflow, we implemented a workaround to force closing connections using a FlowExecutionListener.

Here is the code:

public class AvoidLeakJpaFlowExecutionListener extends JpaFlowExecutionListener {

	private TransactionTemplate transactionTemplate;

	public AvoidLeakJpaFlowExecutionListener(EntityManagerFactory entityManagerFactory,
			PlatformTransactionManager transactionManager) {

		super(entityManagerFactory, transactionManager);

		this.transactionTemplate = new TransactionTemplate(transactionManager);
	}

	@Override
	public void paused(RequestContext context) {

		if (isPersistenceContext(context.getActiveFlow())) {

			final EntityManager em = getEntityManager(context.getFlowExecutionContext().getActiveSession());

			/*
			 * we were not be able to determine if entityManager has opened
			 * connections, then the commit statement is always executed when
			 * flow is paused
			 */

			transactionTemplate.execute(new TransactionCallbackWithoutResult() {
				protected void doInTransactionWithoutResult(TransactionStatus status) {
					/*
					 * Commit using the same connection retrieved by hibernate
					 * lazy load,if exists, otherwise another connection will be
					 * requested from pool
					 */
					em.joinTransaction();
				}
			});

		}

		super.paused(context);
	}

	private boolean isPersistenceContext(FlowDefinition flow) {
		return flow.getAttributes().contains(PERSISTENCE_CONTEXT_ATTRIBUTE);
	}

	private EntityManager getEntityManager(FlowSession session) {
		return (EntityManager) session.getScope().get(PERSISTENCE_CONTEXT_ATTRIBUTE);
	}

}

If anyone has another solution, please, let us know. We did not detect any side effects with this approach.

[spring-operator]

Artem Karpenko commented

Hi Fabricio,

thank you for confirmation and for proposed solution. Interesting enough, our project also had a somewhat similar workaround targeting paused method:

public void paused(RequestContext context) {
    super.paused(context);
    EntityManager entityManager = (EntityManager) context.getFlowScope().get(PERSISTENCE_CONTEXT_ATTRIBUTE);
    if (entityManager != null && entityManager instanceof HibernateEntityManager) {
        HibernateEntityManager hibernateEntityManager = (HibernateEntityManager) entityManager;
        hibernateEntityManager.getSession().disconnect();
    }
}

This was introduced far before I joined our project and seemed to be good enough. Nevertheless we experienced rare cases of choking of application with connection pool being exhausted. Your suggestion made me retest our application and repro-sample once more and I discovered something new. The main thing - that we were using 2.2.1 in production while 2.3.0 fixed some problems related to handling of persistence with subflows; these problems caused additional chaos in my understanding of problem. So I rechecked with 2.3.1 and it appears that your solution (and ours as well) fixes connection leaks in our application.

However I found another case where connections are still leaked. This is not from our real application but I'd post this here so that you would be aware of existing deficiency in presented approach with paused method. You can check out example at https://github.com/artem-karpenko/spring-webflow-issues. I've modified slightly sample project that was added here as repro-project - added extended JPA flow listener, simple subflow configuration and appropriate JSP view. The main point in this case - database connection is leaked (even when using "fixed" listener) when loading of lazy-initialized entities is done using WebFlow's persistence context and this loading is performed during the transition to subflow that does not have configured. Rather specific case, I admit :)

That being said, I myself will probably be satisfied with our "solution" but it'd be nice if this problem would however be addressed because it makes out-of-the-box integration with Hibernate (other JPA providers too?) and any connection pool practically unusable.

[spring-operator]

Fabricio Colombo commented

Hi Artem,

a few days ago we found a very bad side effect with our approach, when the "transactionTemplate.execute" is executed, the transaction is committed before the flow endstate to be achieved.

Since then, we leave aside the problem, and yesterday we started to solve the problem, and our new approach is very similiar.

	@Override
	public void paused(RequestContext context) {
		EntityManager em = getEntityManager(context.getFlowExecutionContext().getActiveSession());

		if (em != null) {
			Session session = em.unwrap(Session.class);

			session.disconnect();
		}

		super.paused(context);
	}

	private EntityManager getEntityManager(FlowSession session) {
		return (EntityManager) session.getScope().get(PERSISTENCE_CONTEXT_ATTRIBUTE);
	}

I agree with you, the approach using session.disconnect() seems good enough.

• The session.disconnect() is only executed if the flow has a persistenceContext (EntityManager != null).
• In this moment, open connection exists only if lazy loading occurred during the flow.

[spring-operator]

guido moscarella commented

I had the same problem, but setting hibernate.connection.release_mode to after_transaction seems to have fixed it. The connection release mode was set to on_close by default. Reading the Hibernate documentation I found out that when set to on_close the connection should be closed when the session is closed or disconnected. And indeed it is closed just after the view is rendered, but if there is some lazy loaded collection hibernate asks for a new connection to load the collection and then it fails to close it! I am wondering if I'm leaking the session instead of the connection now...

[spring-operator]

pas filip commented

Hi, I worked recently on an application that was leaking connections and that was using Webflow + JSF + JPA (1.2). We've had this problem regardless of hibernate version. We tried 4 and 5. We also use hibernate.connection.release_mode = after_transaction but leaks still occur...

The solution we implemented extends Spring jpa flow listener and seems to be working fine so far:

@Override
public void paused(RequestContext context) {
     EntityManager entityManager = getEntityManager(context.getFlowExecutionContext().getActiveSession());
    if (isPersistenceContext(context.getActiveFlow())) {
        unbind(entityManager);
    }
        if ( entityManager == null ){
            return;
        }
        Session session = entityManager.unwrap(Session.class);
        if ( session == null ){
            return;
        }
        session.disconnect();             
}