cromwell icon indicating copy to clipboard operation
cromwell copied to clipboard

Cromwell Failed to summarize metadata

Open chunjie-sam-liu opened this issue 6 years ago • 8 comments

I'm working on mutation calling based on cromwell, the Failed to summarize metadata comes out for several shards in the scatter, then the following processes are aborted. How to fixed this error?

[2018-11-17 09:04:45,38] [info] BackgroundConfigAsyncJobExecutionActor [3df56d2bPreProcessingForVariantDiscovery_GATK4.MarkDuplicates:5:1]: job id: 56011
[2018-11-17 09:04:45,48] [info] BackgroundConfigAsyncJobExecutionActor [3df56d2bPreProcessingForVariantDiscovery_GATK4.MarkDuplicates:5:1]: Status change from - to WaitingForReturnCodeFile
[2018-11-17 09:37:07,47] [error] Failed to summarize metadata
java.sql.SQLTransientConnectionException: db - Connection is not available, request timed out after 3785ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
	at slick.jdbc.hikaricp.HikariCPJdbcDataSource.createConnection(HikariCPJdbcDataSource.scala:14)
	at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
	at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
	at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
	at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
	at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
	at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
	at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-11-17 09:37:14,33] [error] Error summarizing metadata
java.sql.SQLTransientConnectionException: db - Connection is not available, request timed out after 3785ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
	at slick.jdbc.hikaricp.HikariCPJdbcDataSource.createConnection(HikariCPJdbcDataSource.scala:14)
	at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
	at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
	at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
	at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
	at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
	at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
	at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-11-17 09:37:53,75] [warn] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. GET /token Empty -> 200 OK Chunked
[2018-11-17 10:11:19,05] [warn] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. GET /token Empty -> 200 OK Chunked
[Guo-1|12:27:21]

chunjie-sam-liu avatar Nov 18 '18 06:11 chunjie-sam-liu

When I use the cromwell in Local mode, I had same issue, too.

I tried at twice.

In second time, the job work well. I guess the idle time is very long at first time.

hmkim avatar Nov 19 '18 08:11 hmkim

@hmkim I continue the break point to run it again, it works now. What part of process takes long idle time in your instance? what makes the long idle time? In fact, the pipeline always consists of multiple processes and works on hundreds of samples. In case of time, what should i config to avoid this errors not run it again?

chunjie-sam-liu avatar Nov 19 '18 14:11 chunjie-sam-liu

There are many workload in server by non-cromwell job. so, I think it will be solving this issue by defining idle time limitation in cromwell.

hmkim avatar Nov 19 '18 23:11 hmkim

The error occurs again, I read this thread, and configure the local mysql database rather in-memory database.

chunjie-sam-liu avatar Nov 22 '18 05:11 chunjie-sam-liu

@chunjie-sam-liu Thanks to linking thread.

hmkim avatar Nov 24 '18 00:11 hmkim

@chunjie-sam-liu Did you wind up being able to resolve this?

geoffjentry avatar Feb 07 '19 19:02 geoffjentry

@geoffjentry Not really solved. The pipeline could be terminated by the same error, i just extract the samples that are not processed and run it again. It would be better with local MySQL database.

chunjie-sam-liu avatar Feb 12 '19 07:02 chunjie-sam-liu

metadata summary failed ! Have you solved the problem yet?

kevin-furant avatar Dec 01 '21 02:12 kevin-furant