linstor-server
linstor-server copied to clipboard
LINBIT
Unhandled IllegalStateException while restoring node
I just tried to restore evicted node:
root@linstor-controller-7784d94b67-7996f:/# linstor n l
╭─────────────────────────────────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════╡
┊ k8s-eu-de-n01 ┊ SATELLITE ┊ 10.0.0.3:3367 (SSL) ┊ EVICTED ┊
┊ k8s-eu-de-n02 ┊ SATELLITE ┊ 10.0.0.2:3367 (SSL) ┊ Online ┊
┊ k8s-eu-de-n03 ┊ SATELLITE ┊ 10.0.0.4:3367 (SSL) ┊ Online ┊
┊ linstor-controller-7784d94b67-7996f ┊ CONTROLLER ┊ 10.111.0.42:3367 (SSL) ┊ OFFLINE ┊
┊ linstor-controller-7784d94b67-zjht8 ┊ CONTROLLER ┊ 10.111.1.44:3367 (SSL) ┊ Online ┊
╰─────────────────────────────────────────────────────────────────────────────────────╯
root@linstor-controller-7784d94b67-7996f:/# ^C
root@linstor-controller-7784d94b67-7996f:/# ^C
root@linstor-controller-7784d94b67-7996f:/# linstor n rst k8s-eu-de-n01
ERROR:
Exceptions have been converted to responses
Show reports:
linstor error-reports show 62700B54-00000-000059
controller log:
17:26:11.497 [MainWorkerPool-1] ERROR LINSTOR/Controller - SYSTEM - (Node: 'k8s-eu-de-n03') Failed to create meta-data for DRBD volume pvc-66442151-88ee-4769-b547-7c542ee6a279/0 [Report number 62700B54-00000-000065]
17:26:18.737 [MainWorkerPool-1] ERROR LINSTOR/Controller - SYSTEM - (Node: 'k8s-eu-de-n02') Failed to adjust DRBD resource pvc-66442151-88ee-4769-b547-7c542ee6a279 [Report number 62700B54-00000-000066]
17:28:36.930 [grizzly-http-server-0] INFO LINSTOR/Controller - SYSTEM - New volume definition with number '0' of resource definition 'pvc-12bfaa3d-c9e9-4272-990e-5bfde6b82a8e' created.
17:28:37.173 [MainWorkerPool-1] INFO LINSTOR/Controller - SYSTEM - Drbd-auto-verify-Algo for pvc-12bfaa3d-c9e9-4272-990e-5bfde6b82a8e automatically set to crct10dif-pclmul
satellite log:
17:24:45.868 [SSLNetComService] ERROR LINSTOR/Satellite - SYSTEM - Unhandled IllegalStateException [Report number 626FF208-4BE2C-000000]
Error reports:
62700B54-00000-000065
ERROR REPORT 62700B54-00000-000065
============================================================
Application: LINBIT�� LINSTOR
Module: Controller
Version: 1.18.0
Build ID: 648ab925644f53039239c5aec366a11f046f5977
Build time: 2022-04-06T15:53:04+00:00
Error time: 2022-05-02 17:26:11
Node: linstor-controller-7784d94b67-zjht8
Peer: RestClient(10.111.0.40; 'linstor-csi/v0.18.0')
============================================================
Reported error:
===============
Category: RuntimeException
Class name: ApiRcException
Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message: (Node: 'k8s-eu-de-n03') Failed to create meta-data for DRBD volume pvc-66442151-88ee-4769-b547-7c542ee6a279/0
Error context:
(Node: 'k8s-eu-de-n03') Failed to create meta-data for DRBD volume pvc-66442151-88ee-4769-b547-7c542ee6a279/0
ApiRcException entries:
Nr: 1
Message: (Node: 'k8s-eu-de-n03') Failed to create meta-data for DRBD volume pvc-66442151-88ee-4769-b547-7c542ee6a279/0
Asynchronous stage backtrace:
Error has been observed at the following site(s):
|_ checkpoint ? Modify resource-definition
Stack trace:
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
Suppressed exception 1 of 1:
===============
Category: RuntimeException
Class name: OnAssemblyException
Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message:
Error has been observed at the following site(s):
|_ checkpoint ��� Modify resource-definition
Stack trace:
Error context:
(Node: 'k8s-eu-de-n03') Failed to create meta-data for DRBD volume pvc-66442151-88ee-4769-b547-7c542ee6a279/0
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:284
doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:235
lambda$doProcessMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:220
subscribe N reactor.core.publisher.FluxDefer:46
subscribe N reactor.core.publisher.Flux:8357
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243
drainFused N reactor.core.publisher.UnicastProcessor:286
drain N reactor.core.publisher.UnicastProcessor:329
onNext N reactor.core.publisher.UnicastProcessor:408
next N reactor.core.publisher.FluxCreate$IgnoreSink:618
next N reactor.core.publisher.FluxCreate$SerializedSink:153
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:383
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:218
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1128
run N java.util.concurrent.ThreadPoolExecutor$Worker:628
run N java.lang.Thread:829
END OF ERROR REPORT.
62700B54-00000-000066
ERROR REPORT 62700B54-00000-000066
============================================================
Application: LINBIT�� LINSTOR
Module: Controller
Version: 1.18.0
Build ID: 648ab925644f53039239c5aec366a11f046f5977
Build time: 2022-04-06T15:53:04+00:00
Error time: 2022-05-02 17:26:18
Node: linstor-controller-7784d94b67-zjht8
Peer: RestClient(10.111.0.40; 'linstor-csi/v0.18.0')
============================================================
Reported error:
===============
Category: RuntimeException
Class name: ApiRcException
Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message: (Node: 'k8s-eu-de-n02') Failed to adjust DRBD resource pvc-66442151-88ee-4769-b547-7c542ee6a279
Error context:
(Node: 'k8s-eu-de-n02') Failed to adjust DRBD resource pvc-66442151-88ee-4769-b547-7c542ee6a279
ApiRcException entries:
Nr: 1
Message: (Node: 'k8s-eu-de-n02') Failed to adjust DRBD resource pvc-66442151-88ee-4769-b547-7c542ee6a279
Asynchronous stage backtrace:
Error has been observed at the following site(s):
|_ checkpoint ? Modify volume
Stack trace:
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
Suppressed exception 1 of 1:
===============
Category: RuntimeException
Class name: OnAssemblyException
Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message:
Error has been observed at the following site(s):
|_ checkpoint ��� Modify volume
Stack trace:
Error context:
(Node: 'k8s-eu-de-n02') Failed to adjust DRBD resource pvc-66442151-88ee-4769-b547-7c542ee6a279
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:284
doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:235
lambda$doProcessMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:220
subscribe N reactor.core.publisher.FluxDefer:46
subscribe N reactor.core.publisher.Flux:8357
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243
drainFused N reactor.core.publisher.UnicastProcessor:286
drain N reactor.core.publisher.UnicastProcessor:329
onNext N reactor.core.publisher.UnicastProcessor:408
next N reactor.core.publisher.FluxCreate$IgnoreSink:618
next N reactor.core.publisher.FluxCreate$SerializedSink:153
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:383
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:218
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1128
run N java.util.concurrent.ThreadPoolExecutor$Worker:628
run N java.lang.Thread:829
END OF ERROR REPORT.
626FF208-4BE2C-000000
ERROR REPORT 626FF208-4BE2C-000000
============================================================
Application: LINBIT? LINSTOR
Module: Satellite
Version: 1.18.0
Build ID: 648ab925644f53039239c5aec366a11f046f5977
Build time: 2022-04-06T15:53:04+00:00
Error time: 2022-05-02 17:24:45
Node: k8s-eu-de-n01
Peer: 10.0.0.4:4081
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'run', Source file 'TcpConnectorService.java', Line #734
Error message: Unhandled IllegalStateException
Call backtrace:
Method Native Class:Line number
run N com.linbit.linstor.netcom.TcpConnectorService:734
run N java.lang.Thread:829
Caused by:
==========
Category: RuntimeException
Class name: IllegalStateException
Class canonical name: java.lang.IllegalStateException
Generated at: Method 'doHandshake', Source file 'SslTcpConnectorHandshaker.java', Line #103
Error message: com.linbit.linstor.netcom.ssl.SslTcpConnectorService indicates requiring a handshake, but the sun.security.ssl.SSLEngineImpl instance is not in handshake mode
Call backtrace:
Method Native Class:Line number
doHandshake N com.linbit.linstor.netcom.ssl.SslTcpConnectorHandshaker:103
read N com.linbit.linstor.netcom.ssl.SslTcpConnectorPeer:162
run N com.linbit.linstor.netcom.TcpConnectorService:543
run N java.lang.Thread:829
END OF ERROR REPORT.
cc @krakazyabra
