linstor-server
linstor-server copied to clipboard
Published
20 hours ago •
LINBIT
LINBIT
Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference
Hi I use k8s backend. After some days I found that linstor-controller can't start:
LINSTOR, Module Controller
Version: 1.18.0 (648ab925644f53039239c5aec366a11f046f5977)
Build time: 2022-03-28T18:25:25+00:00
Java Version: 11
Java VM: Debian, Version 11.0.14+9-post-Debian-1deb10u1
Operating system: Linux, Version 5.13.0-27-generic
Environment: amd64, 1 processors, 29694 MiB memory reserved for allocations
System components initialization in progress
10:57:32.513 [main] INFO LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
10:57:32.515 [main] INFO LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
10:57:32.533 [Main] INFO LINSTOR/Controller - SYSTEM - Loading API classes started.
10:57:32.800 [Main] INFO LINSTOR/Controller - SYSTEM - API classes loading finished: 267ms
10:57:32.800 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection started.
10:57:32.807 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
10:57:32.807 [Main] INFO LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
10:57:33.297 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection finished: 497ms
10:57:33.436 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
10:57:33.617 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
10:57:33.617 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
10:57:34.577 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
10:57:34.582 [Main] INFO LINSTOR/Controller - SYSTEM - Loading security objects
10:57:34.630 [Main] INFO LINSTOR/Controller - SYSTEM - Current security level is NO_SECURITY
10:57:35.070 [Main] INFO LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
10:57:35.941 [Main] ERROR LINSTOR/Controller - SYSTEM - Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference [Report number 625D441C-00000-000000]
10:57:35.942 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown in progress
10:57:35.943 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
10:57:35.943 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
10:57:35.943 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
10:57:35.945 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
10:57:35.945 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
10:57:35.945 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
10:57:35.945 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown complete
ERROR REPORT 625D441C-00000-000000
============================================================
Application: LINBIT? LINSTOR
Module: Controller
Version: 1.18.0
Build ID: 648ab925644f53039239c5aec366a11f046f5977
Build time: 2022-03-28T18:25:25+00:00
Error time: 2022-04-18 10:57:35
Node: linstor-controller-fcb49b597-z8ht2
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20
Error message: Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference
Call backtrace:
Method Native Class:Line number
ctorNotNull N com.linbit.ErrorCheck:20
<init> N com.linbit.linstor.core.objects.ResourceDefinition:134
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
loadAll N com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
loadAll N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:100
loadAll N com.linbit.linstor.dbdrivers.DatabaseLoader:314
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:176
initialize N com.linbit.linstor.core.DbDataInitializer:108
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:347
main N com.linbit.linstor.core.Controller:585
Caused by:
==========
Category: RuntimeException
Class name: NullPointerException
Class canonical name: java.lang.NullPointerException
Generated at: Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20
Call backtrace:
Method Native Class:Line number
ctorNotNull N com.linbit.ErrorCheck:20
<init> N com.linbit.linstor.core.objects.ResourceDefinition:134
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
loadAll N com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
loadAll N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:100
loadAll N com.linbit.linstor.dbdrivers.DatabaseLoader:314
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:176
initialize N com.linbit.linstor.core.DbDataInitializer:108
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:347
main N com.linbit.linstor.core.Controller:585
END OF ERROR REPORT.
Absolutely the same, with only one exception, that we using older version of LINSTOR:
LINSTOR, Module Controller
Version: 1.17.0 (7e646d83dbbadf1ec066e1bc8b29ae018aff1f66)
Build time: 2021-12-09T07:27:52+00:00
Java Version: 11
Java VM: Debian, Version 11.0.14+9-post-Debian-1deb10u1
Operating system: Linux, Version 4.18.0-80.el8.x86_64
Environment: amd64, 1 processors, 5216 MiB memory reserved for allocations
LINSTOR backend is etcd 3.4.15 To resolve this issue I had to restore by hand deleted etcd keys:
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/ALLOWED_PROVIDER_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/DESCRIPTION
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/DO_NOT_PLACE_WITH_RSC_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/LAYER_STACK
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/NODE_NAME_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/POOL_NAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/POOL_NAME_DISKLESS
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICAS_ON_DIFFERENT
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICAS_ON_SAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICA_COUNT
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/RESOURCE_GROUP_DSP_NAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/UUID
/LINSTOR/SEC_ACL_MAP//resourcegroups/<STORAGE-CLASS-NAME>:PUBLIC/ACCESS_TYPE
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/CREATOR_IDENTITY_NAME
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/OWNER_ROLE_NAME
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/SECURITY_TYPE_NAME
/LINSTOR/VOLUME_GROUPS/<STORAGE-CLASS-NAME>:0/FLAGS
/LINSTOR/VOLUME_GROUPS/<STORAGE-CLASS-NAME>:0/UUID
Possibly, though then I'm not sure how the same issue could apply to the etcd backend.
Hi, I just faced this issue again:
trace log:
LINSTOR, Module Controller
Version: 1.20.0 (9c6f7fad48521899f7a99c564b1d33aeacfdbfa8)
Build time: 2022-11-07T16:37:38+00:00
Java Version: 11
Java VM: Ubuntu, Version 11.0.16+8-post-Ubuntu-0ubuntu122.04
Operating system: Linux, Version 5.15.0-25-generic
Environment: amd64, 1 processors, 15524 MiB memory reserved for allocations
System components initialization in progress
Loading configuration file "/etc/linstor/linstor.toml"
12:54:58.573 [main] INFO LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
12:54:58.575 [main] INFO LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
12:54:58.592 [main] INFO LINSTOR/Controller - SYSTEM - Database type is Kubernetes-CRD
12:54:58.593 [Main] INFO LINSTOR/Controller - SYSTEM - Loading API classes started.
12:54:58.913 [Main] INFO LINSTOR/Controller - SYSTEM - API classes loading finished: 320ms
12:54:58.913 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection started.
12:54:58.920 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
12:54:58.920 [Main] INFO LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
12:54:58.920 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
12:54:58.925 [Main] DEBUG LINSTOR/Controller - SYSTEM - Constructing instance of module "com.linbit.linstor.modularcrypto.JclCryptoModule" with default constructor
12:54:58.925 [Main] INFO LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
12:54:58.925 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
12:54:58.925 [Main] INFO LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
12:54:59.484 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection finished: 571ms
12:54:59.485 [Main] INFO LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
12:54:59.638 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
12:54:59.831 [Main] DEBUG LINSTOR/Controller - SYSTEM - SpaceTrackingService: No instance available to add as a system service
12:54:59.832 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
12:54:59.832 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
12:54:59.833 [Main] INFO LINSTOR/Controller - SYSTEM - Kubernetes-CRD connection URL is "k8s"
12:55:00.721 [Main] TRACE LINSTOR/Controller - SYSTEM - Found database version 9
12:55:00.850 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
12:55:00.854 [Main] INFO LINSTOR/Controller - SYSTEM - Loading security objects
12:55:01.236 [Main] INFO LINSTOR/Controller - SYSTEM - Current security level is NO_SECURITY
12:55:01.236 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/nodesMap)
12:55:01.288 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/nodesMap)
12:55:01.343 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/nodesMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/nodesMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/remoteMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/shutdown)
12:55:01.379 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/CTRLCFG)
12:55:01.471 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=STLTCFG)
12:55:01.472 [Main] INFO LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
12:55:01.472 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all RESOURCE_GROUPSs
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.531 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.531 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/resourcegroups/DFLTRSCGRP)
12:55:01.532 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 1 RESOURCE_GROUPSs
12:55:01.533 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all VOLUME_GROUPSs
12:55:01.589 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 0 VOLUME_GROUPSs
12:55:01.590 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all NODESs
12:55:01.643 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW03)
12:55:01.651 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 13 NODESs
12:55:01.651 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all RESOURCE_DEFINITIONSs
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.718 [Main] ERROR LINSTOR/Controller - SYSTEM - Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference [Report number 63738C22-00000-000000]
12:55:01.719 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown in progress
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'EbsStatusPoll' of type EbsStatusPoll
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'EbsStatusPoll' to complete shutdown
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'ScheduleBackupService' of type ScheduleBackupService
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'ScheduleBackupService' to complete shutdown
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
12:55:01.720 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
12:55:01.721 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
12:55:01.721 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
12:55:01.721 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
12:55:01.721 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown complete
error report:
ERROR REPORT 63738C22-00000-000000
============================================================
Application: LINBIT? LINSTOR
Module: Controller
Version: 1.20.0
Build ID: 9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time: 2022-11-07T16:37:38+00:00
Error time: 2022-11-15 12:55:01
Node: linstor-controller-7dcdfd7cb8-vmkrb
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20
Error message: Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference
Call backtrace:
Method Native Class:Line number
ctorNotNull N com.linbit.ErrorCheck:20
<init> N com.linbit.linstor.core.objects.ResourceDefinition:134
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
loadAll N com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
loadAll N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:102
loadAll N com.linbit.linstor.dbdrivers.DatabaseLoader:330
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:176
initialize N com.linbit.linstor.core.DbDataInitializer:108
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:360
main N com.linbit.linstor.core.Controller:608
Caused by:
==========
Category: RuntimeException
Class name: NullPointerException
Class canonical name: java.lang.NullPointerException
Generated at: Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20
Call backtrace:
Method Native Class:Line number
ctorNotNull N com.linbit.ErrorCheck:20
<init> N com.linbit.linstor.core.objects.ResourceDefinition:134
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
load N com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
loadAll N com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
loadAll N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:102
loadAll N com.linbit.linstor.dbdrivers.DatabaseLoader:330
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:176
initialize N com.linbit.linstor.core.DbDataInitializer:108
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:360
main N com.linbit.linstor.core.Controller:608
END OF ERROR REPORT.
Workaroud is to replace missing resource groups to default one:
rgs=$(kubectl get resourcegroups.internal.linstor.linbit.com -o custom-columns=NAME:.spec.resource_group_name --no-headers)
kubectl get resourcedefinitions.internal.linstor.linbit.com -o custom-columns=NAME:.metadata.name,RG:.spec.resource_group_name --no-headers | \
while read rd rg; do
if ! echo "$rgs" | grep -q "$rg"; then
(
set -x
kubectl patch resourcedefinitions.internal.linstor.linbit.com "$rd" --type=merge -p '{"spec":{"resource_group_name":"DFLTRSCGRP"}}'
)
fi
done
Very strange, I wonder if there is a situation in LINSTOR where we allow deletion of a resource group at the same time as creating a resource definition from it. I'm wondering because it looks like this also happened (happens?) with ETCD, so it may not be related to DB specific code, instead just a check at the wrong time in LINSTOR.
It could be that the CSI driver triggers this issue more frequently, because it tries to delete a resource group during PV deletion (with the assumption that it will fail if resource definitions remain)
I encountered a similar situation in my cluster. The Linstor controller refused to start due to a corrupted database. This happened after a spontaneous restart of the controller, and I couldn't find any other reasons for it. Here is the controller log:
root@linstor-controller-677d7b45d5-sjwn9:/# /usr/bin/piraeus-entry.sh startController
Importing keystore /tmp/tmp.2zp9HsgOUU to /etc/linstor/ssl/keystore.jks...
Entry for alias linstor successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
Certificate was added to keystore
Importing keystore /tmp/tmp.iDuEyryIbX to /etc/linstor/https/keystore.jks...
Entry for alias linstor successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
Certificate was added to keystore
LINSTOR, Module Controller
Version: 1.24.2 (adb19ca96a07039401023410c1ea116f09929295)
Build time: 2023-10-05T13:14:50+00:00
Java Version: 11
Java VM: Ubuntu, Version 11.0.20.1+1-post-Ubuntu-0ubuntu122.04
Operating system: Linux, Version 5.15.0-83-generic
Environment: amd64, 4 processors, 2990 MiB memory reserved for allocations
System components initialization in progress
Loading configuration file "/etc/linstor/linstor.toml"
09:35:33.542 [main] INFO LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
09:35:33.545 [main] INFO LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
09:35:33.618 [main] INFO LINSTOR/Controller - SYSTEM - Database type is Kubernetes-CRD
09:35:33.619 [Main] INFO LINSTOR/Controller - SYSTEM - Loading API classes started.
09:35:34.275 [Main] INFO LINSTOR/Controller - SYSTEM - API classes loading finished: 655ms
09:35:34.275 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection started.
09:35:34.295 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
09:35:34.296 [Main] INFO LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
09:35:34.297 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
09:35:34.307 [Main] INFO LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
09:35:34.308 [Main] INFO LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
09:35:34.309 [Main] INFO LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
09:35:35.603 [Main] INFO LINSTOR/Controller - SYSTEM - Dependency injection finished: 1328ms
09:35:35.604 [Main] INFO LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
09:35:36.053 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
09:35:36.445 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
09:35:36.446 [Main] INFO LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
09:35:36.447 [Main] INFO LINSTOR/Controller - SYSTEM - Kubernetes-CRD connection URL is "k8s"
09:35:38.424 [Main] INFO LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
09:35:38.437 [Main] INFO LINSTOR/Controller - SYSTEM - Security objects load from database is in progress
09:35:39.015 [Main] INFO LINSTOR/Controller - SYSTEM - Security objects load from database completed
09:35:39.016 [Main] INFO LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
09:35:39.765 [Main] ERROR LINSTOR/Controller - SYSTEM - Unknown error during loading data from DB [Report number 651FD4E4-00000-000000]
09:35:39.767 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown in progress
09:35:39.771 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'EbsStatusPoll' of type EbsStatusPoll
09:35:39.772 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'EbsStatusPoll' to complete shutdown
09:35:39.773 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'ScheduleBackupService' of type ScheduleBackupService
09:35:39.774 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'ScheduleBackupService' to complete shutdown
09:35:39.775 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
09:35:39.775 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
09:35:39.776 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
09:35:39.787 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
09:35:39.789 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
09:35:39.789 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
09:35:39.789 [Thread-2] INFO LINSTOR/Controller - SYSTEM - Shutdown complete
Error report:
root@linstor-controller-677d7b45d5-sjwn9:/# linstor error-reports 651FD4E4-00000-000000
usage: linstor error-reports [-h] {delete, list, show} ...
linstor error-reports: error: argument {delete, list, show}: invalid choice: '651FD4E4-00000-000000' (choose from 'list', 'l', 'show', 's', 'delete', 'd')
root@linstor-controller-677d7b45d5-sjwn9:/# linstor error-reports show 651FD4E4-00000-000000
^C
linstor: Client exiting (received SIGINT)
root@linstor-controller-677d7b45d5-sjwn9:/# cd /var/log
root@linstor-controller-677d7b45d5-sjwn9:/var/log# cd linstor-controller/
root@linstor-controller-677d7b45d5-sjwn9:/var/log/linstor-controller# ls -l
total 32
-rw-r--r-- 1 root root 3511 Oct 6 09:35 ErrorReport-651FD4E4-00000-000000.log
-rw-r--r-- 1 root root 20480 Oct 6 09:35 error-report.mv.db
-rw-r--r-- 1 root root 4575 Oct 6 09:35 linstor-Controller.log
root@linstor-controller-677d7b45d5-sjwn9:/var/log/linstor-controller# cat ErrorReport-651FD4E4-00000-000000.log
ERROR REPORT 651FD4E4-00000-000000
============================================================
Application: LINBIT? LINSTOR
Module: Controller
Version: 1.24.2
Build ID: adb19ca96a07039401023410c1ea116f09929295
Build time: 2023-10-05T13:14:50+00:00
Error time: 2023-10-06 09:35:39
Node: linstor-controller-677d7b45d5-sjwn9
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'loadCoreObjects', Source file 'DatabaseLoader.java', Line #680
Error message: Unknown error during loading data from DB
Call backtrace:
Method Native Class:Line number
loadCoreObjects N com.linbit.linstor.dbdrivers.DatabaseLoader:680
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:169
initialize N com.linbit.linstor.core.DbDataInitializer:101
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:361
main N com.linbit.linstor.core.Controller:609
Caused by:
==========
Description:
ObjProt (/resourcegroups/SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7) not found!
Category: LinStorException
Class name: DatabaseException
Class canonical name: com.linbit.linstor.dbdrivers.DatabaseException
Generated at: Method 'getInstance', Source file 'ObjectProtectionFactory.java', Line #89
Error message: ObjProt (/resourcegroups/SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7) not found!
ErrorContext:
Call backtrace:
Method Native Class:Line number
getInstance N com.linbit.linstor.security.ObjectProtectionFactory:89
getObjectProtection N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:271
load N com.linbit.linstor.core.objects.ResourceGroupDbDriver:302
load N com.linbit.linstor.core.objects.ResourceGroupDbDriver:59
loadAll N com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:237
loadAll N com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170
loadCoreObjects N com.linbit.linstor.dbdrivers.DatabaseLoader:334
loadCoreObjects N com.linbit.linstor.core.DbDataInitializer:169
initialize N com.linbit.linstor.core.DbDataInitializer:101
startSystemServices N com.linbit.linstor.core.ApplicationLifecycleManager:87
start N com.linbit.linstor.core.Controller:361
main N com.linbit.linstor.core.Controller:609
END OF ERROR REPORT.
Meanwhile, the RG SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7 existed among the resources. Before making any changes, I saved all Linstor resources in yaml format and can provide them upon request. For now, I decided to temporarily transfer all resources associated with this RG to DFLTGRP and then change it to another group. The controller wouldn't start until this RG was completely cleared.
