HashDataInc
bireme启动后,1分钟左右就出现假死情况,日志没有记录任何错误
Please check if there is active load query in database first. And then check loading status with rest api or jmx.
@wangzw 第一句话什么意思？我没有理解。。我同步5张表没有问题，表一增加 就会出现上述情况
rest api ip:8080/mysql
result:
rest api ip:8080/
result:
why?
@wangzw 发现我只要是 在bireme里面配置的表多了，就会出现此问题，，例如 kafka里面只有5张表的数据，而bireme里面配置50张表，就会出问题。是不是需要合理调节下参数？
@jinxiaoxin Thanks for your information.
It would be better to enable TRACE level logging for bireme in log2j.xml.
@Rucfisher What if the topic we want to subscribe does not exist? Need more investigation.
@jinxiaoxin Please also change the root level of log to trace level and try again.
@jinxiaoxin It is also helpful to print stack of bireme using command line jstack
Any message would help. Please paste to this issue or mail me by email. jstack information would be also useful.
@wangzw jstack info： jstack -l pid jstack_message.txt
能看出什么问题吗？ 我突然又发现如果kafka中未消费数据过多~ 启动bireme 基本会在1分钟左右就假死~
"Provider mysql" #38 prio=5 os_prio=0 tid=0x0000000001e16000 nid=0x8e9 waiting on condition [0x00007fefd4a3e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfd06170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:385)
at cn.hashdata.bireme.provider.KafkaProvider.call(KafkaProvider.java:161)
at cn.hashdata.bireme.provider.KafkaProvider.call(KafkaProvider.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfc967e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Dispatcher" #37 prio=5 os_prio=0 tid=0x0000000001d0d800 nid=0x8e8 waiting for monitor entry [0x00007fefd4d3e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at cn.hashdata.bireme.RowCache.createBatch(RowCache.java:91)
- waiting to lock <0x00000003cfe7c4e0> (a cn.hashdata.bireme.RowCache)
at cn.hashdata.bireme.RowCache.addRows(RowCache.java:62)
at cn.hashdata.bireme.Dispatcher.insertRowSet(Dispatcher.java:139)
at cn.hashdata.bireme.Dispatcher.checkTansformResults(Dispatcher.java:124)
at cn.hashdata.bireme.Dispatcher.call(Dispatcher.java:73)
at cn.hashdata.bireme.Dispatcher.call(Dispatcher.java:34)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfce25f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"TaskGenerator" #36 prio=5 os_prio=0 tid=0x0000000001d0b800 nid=0x8e7 waiting on condition [0x00007fefd4e40000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfe7c768> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:385)
at cn.hashdata.bireme.RowCache.createBatch(RowCache.java:116)
- locked <0x00000003cfe7c4e0> (a cn.hashdata.bireme.RowCache)
at cn.hashdata.bireme.RowCache.fetchBatch(RowCache.java:130)
at cn.hashdata.bireme.TaskGenerator.generateMergeTask(TaskGenerator.java:96)
at cn.hashdata.bireme.TaskGenerator.call(TaskGenerator.java:63)
at cn.hashdata.bireme.TaskGenerator.call(TaskGenerator.java:32)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfd54b30> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Seems like a dead lock
Remove this line may work. And we will finally fix this issue in 2.0 release.
https://github.com/HashDataInc/bireme/blob/e5e388911fe0dbddb14b896a3ba398ba0d0241f1/src/main/java/cn/hashdata/bireme/RowCache.java#L130
1、注释掉了createBatch(); 还是同样的情况，kafka数据堆积过多，就会导致1分钟之内就会假死。
2、为什么我的HTTP服务器一直报500错误
