charts icon indicating copy to clipboard operation
charts copied to clipboard

[bitnami/ejbca] Upon each update of EJBCA, the application attempts to reinstall itself, resulting in an error.

Open siddjellali opened this issue 1 year ago • 2 comments

Name and Version

bitnami/ejbca: 8.2.0-1-debian-12-r12

What architecture are you using?

amd64

What steps will reproduce the bug?

  • Update EJBCA to the latest version.
  • Observe the logs for the mentioned error during the update process.

Are you using any custom parameters or values?

I deploy configuration through fluxcd :

 values:
  # https://github.com/bitnami/charts/blob/main/bitnami/ejbca/values.yaml
    image:
      tag: 8.2.0-1-debian-12-r12
    ejbcaAdminUsername: xxxxxxx
    ejbcaAdminPassword: xxxxxxx
    mariadb:
      auth:
        rootPassword: xxxxxxx
        database: xxxxxxx
        username: xxxxxxx
        password: xxxxxxx

What is the expected behavior?

Update successful

What do you see instead?

pod failed due to error message

Additional information

ejbca 12:20:33.30 INFO  ==> 
ejbca 12:20:33.30 INFO  ==> Welcome to the Bitnami ejbca container
ejbca 12:20:33.31 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ejbca 12:20:33.31 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ejbca 12:20:33.32 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
ejbca 12:20:33.32 INFO  ==> 
ejbca 12:20:33.35 INFO  ==> ** Starting ejbca setup **
ejbca 12:20:33.38 WARN  ==> Skipping export of 'EJBCA_WILDFLY_ADMIN_PASSWORD'. '/opt/bitnami/wildfly/tmp/wildfly_admin.pwd' is not readable.
ejbca 12:20:33.39 INFO  ==> Validating settings in EJBCA_* env vars...
ejbca 12:20:33.39 INFO  ==> Initializing EJBCA...
ejbca 12:20:33.42 INFO  ==> Deploying EJBCA from scratch
ejbca 12:20:33.50 INFO  ==> Creating database tables and indexes
ERROR 1050 (42S01) at line 1: Table 'AccessRulesData' already exists

siddjellali avatar Jun 15 '24 12:06 siddjellali

Any idea guys ?

siddjellali avatar Jun 21 '24 22:06 siddjellali

Hi, thank you for opening this issue. It looks like the installation is not detecting previous persisted data (via PVCs) associated to EJBCA so it is trying to install the deployment from scratch:

ejbca 12:20:33.42 INFO  ==> Deploying EJBCA from scratch

However, the database seems already populated, hence the conflict.

andresbono avatar Jun 24 '24 12:06 andresbono

Exactly... Any workarround ? Do you need anything from me ?

siddjellali avatar Jun 24 '24 14:06 siddjellali

Do you want to keep the database information or can you drop it to start from scratch? Note that any workaround you try could lead to inconsistencies in the application state.

andresbono avatar Jul 03 '24 08:07 andresbono

I can delete everything and start with a fresh install, but will I lose everything again with the next update? Have you found the bug?

siddjellali avatar Jul 03 '24 20:07 siddjellali

A version upgrade should not make your persisted data to be removed. Something else should have happened... Also, please take into account the following:

  • Read the Upgrading notes when upgrading major versions.
  • Be sure you have backups of your persisted data so you can restore in case of eventual loss.

andresbono avatar Jul 04 '24 09:07 andresbono

Sure, but do you know where the problem lies? Is it with the Bitnami container or the Helm chart, or is it directly related to EJBCA?

siddjellali avatar Jul 04 '24 18:07 siddjellali

but do you know where the problem lies?

No, I honestly don't know. An upgrade should not remove the PVCs of a Helm installation. I don't know the specificities of fluxcd though.

If you are able to reproduce this again, please, give us more details of the specific scenario you are testing (original versions and values, target versions and values...) so we can try to reproduce it on our side.

andresbono avatar Jul 05 '24 07:07 andresbono

The problem is in the file libejbca.sh, line 617: Depending on the state of the local wildfly it's decided to assume that the database is initialized or not. That's not necessarily true, as siddjellali already pointed out.

An approach to fix it could be to surround the sql scripts executed in line 369 / 370 with a 'IF NOT EXISTS' . Or try to detect in line 369 / 370 whether the execution of the scripts in necessary.

kuehne-trustable-de avatar Jul 15 '24 15:07 kuehne-trustable-de

Thank you for your suggestion, @kuehne-trustable-de. But wouldn't it be inconsistent to have the database already initialized but not the data directories?

https://github.com/bitnami/containers/blob/6e9a6dcfeaea3e60800b8f029922a1f8e2eb9df7/bitnami/ejbca/8/debian-12/rootfs/opt/bitnami/scripts/ejbca-env.sh#L144

In any case, please feel free to send a PR if you consider that the initialization logic can be improved!

andresbono avatar Jul 17 '24 15:07 andresbono

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Aug 02 '24 01:08 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Aug 07 '24 01:08 github-actions[bot]

Up ! Could you please reopen the issue ?

siddjellali avatar Aug 10 '24 10:08 siddjellali

Please @kuehne-trustable-de, @siddjellali, check my last comment: https://github.com/bitnami/charts/issues/27186#issuecomment-2233563522

andresbono avatar Aug 12 '24 11:08 andresbono

Yes, starting with just a database with the content of the last installation makes no sense. The best solution would be to keep both the PVCs backing the database content and the corresponding data directories. As far as I know the data dictionaries do not survive an 'uninstall', in contrast to the database data. But I did not figure where to change the persistence mode of the data directories.

kuehne-trustable-de avatar Aug 12 '24 12:08 kuehne-trustable-de

The most straightforward option would be to add an annotation in the PVCs to kept them on a helm uninstall (docs):

persistence:
  annotations:
    helm.sh/resource-policy: keep

The difference with the database PVC is that the database one is created from a volumeClaimTemplates in a StatefulSet, so it is not managed by Helm at all.

You could also play with your storageclasses and PVs to set the reclaimPolicy / persistentVolumeReclaimPolicy to Retain.

andresbono avatar Aug 19 '24 10:08 andresbono

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Sep 04 '24 01:09 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Sep 09 '24 01:09 github-actions[bot]

For the benefit to solve this at some point, here some clues.

Before I get this error:

ejbca 08:03:57.52 INFO  ==> 
ejbca 08:03:57.53 INFO  ==> Welcome to the Bitnami ejbca container
ejbca 08:03:57.53 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ejbca 08:03:57.53 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ejbca 08:03:57.53 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
ejbca 08:03:57.53 INFO  ==> 
ejbca 08:03:57.54 INFO  ==> ** Starting ejbca setup **
ejbca 08:03:57.55 WARN  ==> Skipping export of 'EJBCA_WILDFLY_ADMIN_PASSWORD'. '/opt/bitnami/wildfly/tmp/wildfly_admin.pwd' is not readable.
ejbca 08:03:57.55 INFO  ==> Validating settings in EJBCA_* env vars...
ejbca 08:03:57.56 INFO  ==> Initializing EJBCA...
ejbca 08:03:57.56 INFO  ==> Deploying EJBCA from scratch
ejbca 08:04:17.75 INFO  ==> Creating database tables and indexes
ejbca 08:04:23.19 INFO  ==> Creating wildfly management user...
ejbca 08:04:24.41 INFO  ==> Starting wildfly...
ejbca 08:05:24.18 INFO  ==> Creating data source
ejbca 08:05:55.57 INFO  ==> Configure WildFly Remoting
ejbca 08:06:33.42 INFO  ==> Configure logging
ejbca 08:07:03.33 INFO  ==> Remove the ExampleDS DataSource
ejbca 08:07:27.13 INFO  ==> Configure email
ejbca 08:07:59.62 INFO  ==> Configure redirection
ejbca 08:08:34.11 INFO  ==> Deploying EJBCA application
ejbca 08:11:04.52 ERROR ==> Timeout deploying /opt/bitnami/ejbca/dist/ejbca.ear to WildFly: File /opt/bitnami/wildfly/standalone/deployments/ejbca.ear.deployed was not generated

I also noticed that if I remove all probes from the deployment, I have less troubles. As I am testing with very low resources, I assume that kubernetes and possibly also wildfly have to strict timeout.

rriemann avatar Oct 17 '24 08:10 rriemann