Devicehive server crash after runing a long time

[chensohg]

I am running a devicehive server 3.5.0 on CentOS 7. and I am getting the following error after runing a long time and have to restart the server. Could you please help to have a look and advice for possible cause and solution? Thank you,

2019-08-05 09:49:34.148 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/info ] 2019-08-05 09:49:34.177 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/info ] 2019-08-05 09:49:34.180 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/info ] 2019-08-05 09:49:34.204 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/info ] 2019-08-05 09:49:34.214 [XNIO-2 task-5] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/token ] 2019-08-05 09:49:34.231 [XNIO-2 task-5] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/token ] 2019-08-05 09:49:34.233 [XNIO-2 task-6] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ POST /dh/rest/token ] 2019-08-05 09:49:34.243 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL - select user0_.id as id1_3_, user0_.all_device_types_available as all_devi2_3_, user0_.data as data3_3_, user0_.intro_reviewed as intro_re4_3_, user0_.last_login as last_log5_3_, user0_.login as login6_3_, user0_.login_attempts as login_at7_3_, user0_.password_hash as password8_3_, user0_.password_salt as password9_3_, user0_.role as role10_3_, user0_.status as status11_3_ from "dh_user" user0_ where user0_.login=? and user0_.status<>3 2019-08-05 09:49:34.247 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL - select configurat0_.name as name1_0_, configurat0_.entity_version as entity_v2_0_, configurat0_.value as value3_0_ from configuration configurat0_ where configurat0_.name=? 2019-08-05 09:49:34.249 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL - update "dh_user" set all_device_types_available=?, data=?, intro_reviewed=?, last_login=?, login=?, login_attempts=?, password_hash=?, password_salt=?, role=?, status=? where id=? 2019-08-05 09:49:34.272 [XNIO-2 task-6] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ POST /dh/rest/token ] 2019-08-05 09:49:34.280 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/token/create ] 2019-08-05 09:49:34.285 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/token/create ] 2019-08-05 09:49:34.287 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ POST /dh/rest/token/create ] 2019-08-05 09:49:34.332 [XNIO-2 task-8] DEBUG org.hibernate.SQL - select user0_.id as id1_3_0_, user0_.all_device_types_available as all_devi2_3_0_, user0_.data as data3_3_0_, user0_.intro_reviewed as intro_re4_3_0_, user0_.last_login as last_log5_3_0_, user0_.login as login6_3_0_, user0_.login_attempts as login_at7_3_0_, user0_.password_hash as password8_3_0_, user0_.password_salt as password9_3_0_, user0_.role as role10_3_0_, user0_.status as status11_3_0_ from "dh_user" user0_ where user0_.id=? 2019-08-05 09:49:34.343 [XNIO-2 task-8] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission MANAGE_TOKEN 2019-08-05 09:49:34.383 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ POST /dh/rest/token/create ] 2019-08-05 09:49:35.651 [XNIO-2 task-9] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/user/current ] 2019-08-05 09:49:35.655 [XNIO-2 task-9] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/user/current ] 2019-08-05 09:49:35.657 [XNIO-2 task-10] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/user/current ] 2019-08-05 09:49:35.661 [XNIO-2 task-10] DEBUG org.hibernate.SQL - select user0_.id as id1_3_0_, user0_.all_device_types_available as all_devi2_3_0_, user0_.data as data3_3_0_, user0_.intro_reviewed as intro_re4_3_0_, user0_.last_login as last_log5_3_0_, user0_.login as login6_3_0_, user0_.login_attempts as login_at7_3_0_, user0_.password_hash as password8_3_0_, user0_.password_salt as password9_3_0_, user0_.role as role10_3_0_, user0_.status as status11_3_0_ from "dh_user" user0_ where user0_.id=? 2019-08-05 09:49:35.683 [XNIO-2 task-10] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission GET_CURRENT_USER 2019-08-05 09:49:35.693 [XNIO-2 task-10] DEBUG admin org.hibernate.SQL - select user0_.id as id1_3_0_, network2_.id as id1_4_1_, user0_.all_device_types_available as all_devi2_3_0_, user0_.data as data3_3_0_, user0_.intro_reviewed as intro_re4_3_0_, user0_.last_login as last_log5_3_0_, user0_.login as login6_3_0_, user0_.login_attempts as login_at7_3_0_, user0_.password_hash as password8_3_0_, user0_.password_salt as password9_3_0_, user0_.role as role10_3_0_, user0_.status as status11_3_0_, network2_.description as descript2_4_1_, network2_.entity_version as entity_v3_4_1_, network2_.name as name4_4_1_, networks1_.user_id as user_id2_7_0__, networks1_.network_id as network_1_7_0__ from "dh_user" user0_ left outer join user_network networks1_ on user0_.id=networks1_.user_id left outer join network network2_ on networks1_.network_id=network2_.id where user0_.id=? 2019-08-05 09:49:35.712 [XNIO-2 task-10] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/user/current ] 2019-08-05 09:49:35.750 [XNIO-2 task-11] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/network ] 2019-08-05 09:49:35.754 [XNIO-2 task-11] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/network ] 2019-08-05 09:49:35.756 [XNIO-2 task-12] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/network ] 2019-08-05 09:49:35.792 [XNIO-2 task-12] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission GET_NETWORK 2019-08-05 09:49:35.806 [XNIO-2 task-12] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/network ] 2019-08-05 09:49:35.813 [pool-1-thread-2] ERROR c.l.disruptor.FatalExceptionHandler - Exception processing: 11 com.devicehive.model.ServerEvent@22db5968 java.lang.NullPointerException: null at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[disruptor-3.3.6.jar!/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191] Exception in thread "pool-1-thread-2" java.lang.RuntimeException: java.lang.NullPointerException at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:165) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ... 3 more 2019-08-05 09:49:37.346 [XNIO-2 task-13] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/user ] 2019-08-05 09:49:37.350 [XNIO-2 task-13] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/user ] 2019-08-05 09:49:37.352 [XNIO-2 task-14] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/user ] 2019-08-05 09:49:37.362 [XNIO-2 task-14] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission MANAGE_USER Exception in thread "pool-1-thread-1" java.lang.RuntimeException: java.lang.NullPointerException2019-08-05 09:49:37.367 [pool-1-thread-1] ERROR c.l.disruptor.FatalExceptionHandler - Exception processing: 12

com.devicehive.model.ServerEvent@1d0f3cfc java.lang.NullPointerException: null at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[disruptor-3.3.6.jar!/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:165)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ... 3 more 2019-08-05 09:49:37.369 [XNIO-2 task-14] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/user ]