Failed to install with Helm and Percona

[mpavlovicbb]

We have tryed to install Piraeus with MariaDB database instead of ETCD. Installation with ETCD passed without any problem.

We have tryed various commands and users, notnihg worked. Commads that we tryed are: helm install -n linstor piraeus-op ./charts/piraeus --set "operator.controller.dbConnectionURL=jdbc:mysql://user:[email protected]:3306/linstor?createDatabaseIfNotExist=true&useMysqlMetadata=true" and variations on this were

• --set "operator.controller.dbConnectionURL=jdbc:mariadb://user:[email protected]/linstor?createDatabaseIfNotExist=true"
• --set "operator.controller.dbConnectionURL=jdbc:mariadb://devops-haproxy.mysql-ha/linstor?user=user&password=pass" And more variations to this like adding and removing port, moving username and password from the end to the begining of line, etc. Etcd is disabled in values.yaml.

Here is log from cs-controller:

I0129 12:20:12.423886       1 leaderelection.go:242] attempting to acquire leader lease  linstor/piraeus-op-cs...

time="2021-01-29T12:20:12Z" level=info msg="long live our new leader: 'piraeus-op-cs-controller-7d99c4fff5-vv4bx'!"

I0129 12:20:42.998653       1 leaderelection.go:252] successfully acquired lease linstor/piraeus-op-cs

time="2021-01-29T12:20:42Z" level=info msg="long live our new leader: 'piraeus-op-cs-controller-7d99c4fff5-zvm6w'!"

time="2021-01-29T12:20:43Z" level=info msg="starting command '/usr/bin/piraeus-entry.sh' with arguments: '[startController]'"

LINSTOR, Module Controller

Version:            1.11.1 (fe95a94d86c66c6c9846a3cf579a1a776f95d3f4)

Build time:         2021-01-13T08:34:55+00:00

Java Version:       11

Java VM:            Debian, Version 11.0.9.1+1-post-Debian-1deb10u2

Operating system:   Linux, Version 5.4.0-64-generic

Environment:        amd64, 1 processors, 1925 MiB memory reserved for allocations

System components initialization in progress

12:20:43.951 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.

12:20:43.953 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'

12:20:43.988 [main] WARN  io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN, Sentry operations will do nothing! See documentation: https://docs.sentry.io/clients/java/ ***

12:20:43.999 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.

12:20:44.332 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 332ms

12:20:44.332 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.

12:20:44.344 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"

12:20:44.345 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" was successful

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/linstor-server/lib/guice-4.2.3.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

12:20:45.308 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 976ms

12:20:45.604 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem

12:20:45.717 [Main] INFO  LINSTOR/Controller - SYSTEM - SpaceTracking using SQL driver

12:20:45.719 [Main] INFO  LINSTOR/Controller - SYSTEM - SpaceTrackingService: Instance added as a system service

12:20:45.720 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService

12:20:45.721 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the database connection pool

12:20:45.767 [Main] ERROR LINSTOR/Controller - SYSTEM - Database initialization error [Report number 6013FD9B-00000-000000]

12:20:45.774 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress

12:20:45.778 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'SpaceTrackingService' of type SpaceTrackingService

12:20:45.780 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'SpaceTrackingService' to complete shutdown

12:20:45.780 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService

12:20:45.781 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown

12:20:45.781 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'DatabaseService' of type DatabaseService

12:20:45.782 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'DatabaseService' to complete shutdown

12:20:45.782 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService

12:20:45.783 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown

12:20:45.784 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown complete

time="2021-01-29T12:20:45Z" level=fatal msg="failed to run" err="exit status 20" ```

[WanzenBug]

You want to use:

operator.controller.dbConnectionURL=jdbc:mysql://devops-haproxy.mysql-ha:3306/LINSTOR?createDatabaseIfNotExist=true&useMysqlMetadata=true&user=user&password=pass

Note that the database name is LINSTOR instead of the lowercase linstor, second change is adding the username and password to query string. There seems to be an issue with the jdbc string parser in linstor when using the user:pass@host scheme.

[mpavlovicbb]

I am sure that we tryed it before. But, we tryed it again today and again same error as described in firsrt post. Before instalation we have created "LINSTOR" DB.

[WanzenBug]

Very strange, i just tested it with this setting:

jdbc:mysql://mariadb.mariadb.svc:3306/LINSTOR?createDatabaseIfNotExist=true&useMysqlMetadata=true&user=root&password=<mypassword>

and it seemed to work fine.

To investigate further, you can, after installation:

1. temporarily stop the operator: kubectl scale deployment piraeus-op-operator --replicas=0
2. Edit the controller deployment kubectl edit deployment piraeus-op-cs-controller a) replace the args: [startController] setting with command: [tail, -f, /dev/null]. This will start the container without starting the controller b) Remove the livenessProbe section, as it would cause the container to restart after a minute or so
3. Once the controller container is running, use kubectl exec -it deployment/piraeus-op-cs -- bash to get a shell in the container and run /entry.sh startController. This will start the controller, which will hopefully print similar logs to above. Once its encountered the error and stopped, you can check the file /var/log/linstor-controller/ErrorReport-<something>. The <something> will be the ID printed in the controller logs. That report should show more information on what exactly went wrong.

[mpavlovicbb]

Just one coorection of investigation proces, /entry.sh is actualy /usr/bin/piraeus-entry.sh.

We did as described and output is following:

Description:
    Database initialization error

Category:                           LinStorException
Class name:                         SystemServiceStartException
Class canonical name:               com.linbit.SystemServiceStartException
Generated at:                       Method 'initialize', Source file 'DbConnectionPoolInitializer.java', Line #56

Error message:                      Database initialization error


Caused by:
==========

Category:                           RuntimeException
Class name:                         LinStorDBRuntimeException
Class canonical name:               com.linbit.linstor.LinStorDBRuntimeException
Generated at:                       Method 'setTransactionIsolation', Source file 'DbConnectionPool.java', Line #485

Error message:                      Failed to set transaction isolation

Caused by:
==========

Category:                           Exception
Class name:                         SQLException
Class canonical name:               java.sql.SQLException
Generated at:                       Method 'getConnection', Source file 'PoolingDataSource.java', Line #142

Error message:                      Cannot get a connection, pool error Unable to activate object

Caused by:
==========

Category:                           RuntimeException
Class name:                         NoSuchElementException
Class canonical name:               java.util.NoSuchElementException
Generated at:                       Method 'borrowObject', Source file 'GenericObjectPool.java', Line #462

Error message:                      Unable to activate object

Caused by:
==========

Category:                           Exception
Class name:                         SQLTransientConnectionException
Class canonical name:               java.sql.SQLTransientConnectionException
Generated at:                       Method 'createException', Source file 'ExceptionFactory.java', Line #79

Error message:                      (conn=180099) Percona-XtraDB-Cluster doesn't recommend using SERIALIZABLE isolation with pxc_strict_mode = ENFORCING

Afther this, for test we changed strict mode from enforcing to permissive on Percona and got following error: Error message: (conn=444648) Specified key was too long; max key length is 3072 bytes

Anyway, Percona docs recomends strict mode to be enforcing and our cluster is setted up as they recommended.

If you have any other suggestion or you want to investigate further or to support Percona we will be happy to help.

[WanzenBug]

/usr/bin/piraeus-entry.sh

Yeah, sorry about that, I was testing with a slight different image.

I guess you should head over to https://github.com/linbit/linstor-server since this is not really a Piraeus issue but an issue with LINSTOR.