infinispan-operator icon indicating copy to clipboard operation
infinispan-operator copied to clipboard

Published 20 hours ago • verified publisher icon infinispan

Access denied on /opt/infinispan/server/data/___global.lck.

Open mxandeco opened this issue 4 years ago • 35 comments

Following the manual install tutorial from https://github.com/infinispan/infinispan-simple-tutorials/tree/master/operator.

Tried versions 1.1.0 and 1.1.1, deploying on EKS i can confirm both pv and pvc were created properly, the pods won't start, and I can see the logs:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/opt/infinispan/bin/config-generator.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
19:26:22,229 INFO  (main) [BOOT] JVM OpenJDK 64-Bit Server VM Oracle Corporation 11.0.7+10-LTS
19:26:22,233 INFO  (main) [BOOT] JVM arguments = -Xmx200M -Xms200M -XX:MaxRAM=420M -Dsun.zip.disableMemoryMapping=true -XX:+UseSerialGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:+ExitOnOutOfMemoryError -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=96m -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/opt/infinispan
19:26:22,238 INFO  (main) [BOOT] PID = 94
19:26:22,557 INFO  (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
19:26:22,558 INFO  (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan/server/conf/infinispan.xml
19:26:22,558 INFO  (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
19:26:24,952 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
19:26:25,016 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
19:26:25,019 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
19:26:28,724 INFO  (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.7.Final
19:26:28,726 WARN  (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
19:26:28,816 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:139)
	at org.infinispan.server.tool.Main.run(Main.java:98)
	at org.infinispan.server.Bootstrap.main(Bootstrap.java:39)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.infinispan.server.loader.Loader.main(Loader.java:71)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:760)
	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:726)
	at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:51)
	at org.infinispan.security.Security.doPrivileged(Security.java:46)
	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:31)
	at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:54)
	at org.infinispan.server.Server.run(Server.java:286)
	... 8 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:87)
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:64)
	at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
	at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:587)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:578)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:341)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:237)
	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:755)
	... 14 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
	at java.base/java.io.FileOutputStream.open0(Native Method)
	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:81)
	... 25 more

19:26:28,819 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
19:26:28,832 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped

mxandeco avatar May 12 '20 19:05 mxandeco

I think is related to this: https://github.com/infinispan/infinispan/pull/8231 Atm workaround in this situation is cleanup the /opt/infinispan/server/data folder in the PV from any lock file before starting Infinispan.

rigazilla avatar May 13 '20 08:05 rigazilla

Hi @rigazilla Im having the issue with brand new created infinispan crd, there is no "old" lock files in dir.

I have mounted the volume to a "dummy" pod and changed the fs permissions, to 777 as a test, and then infinispan pod started just fine. But I would have to do that for every new replica because a new pv is created =/.

I can't find an option to configure the security-context, custom pv or init container that i could use to fix the permissions.

Any other suggestions in how could i fix this without having to manually entering the volume and changing the permissions for each new replica?

Thanks

mxandeco avatar May 13 '20 11:05 mxandeco

@mxandeco how would you configure the security-context in a deployment/pod? It would be great to have a feature for that allows to do that also in the infinispan CR.

In the meanwhile as a hack workaround you can try adding this entry in the env list of the operator.yaml:

     - name: MAKE_DATADIR_WRITABLE
       value: "true"

this should start a busybox init container that just set writable access to the data dir.

rigazilla avatar May 13 '20 13:05 rigazilla

@rigazilla The MAKE_DATADIR_WRITABLE worked fine for me, maybe would be great to get that documented?

Regarding the:

  • security context. I thought I could try change the runAsUser or fsGroup to get the proper permission to write to the mounted volumes.
  • custom init, I thought i could write a init container to do what the init container is doing when MAKE_DATADIR_WRITABLE is enabled.

Thanks for the support.

mxandeco avatar May 14 '20 12:05 mxandeco

@rigazilla The MAKE_DATADIR_WRITABLE worked fine for me, maybe would be great to get that documented?

I would prefer to have a clean solution for this, i.e. setting context correctly or understand which are the conditions that lead to a readonly pv. but yeah, in the meanwhile it would be good to have it documented, since seems to be the only available solution atm.

rigazilla avatar May 14 '20 14:05 rigazilla

@mxandeco How did you add MAKE_DATADIR_WRITABLE to env section of operator.yml? I tried to change the file in the local and applied the spec on a fresh operator installation. This does applies the new env in the operators pod but it magically vanishes away as 'something' restarts the replicaset of the operators. I am getting it is being overrided by the original spec the git repo. I am sure I am missing something here.

K890 avatar Mar 19 '21 07:03 K890

@mxandeco How did you add MAKE_DATADIR_WRITABLE to env section of operator.yml? I tried to change the file in the local and applied the spec on a fresh operator installation. This does applies the new env in the operators pod but it magically vanishes away as 'something' restarts the replicaset of the operators. I am getting it is being overrided by the original spec the git repo. I am sure I am missing something here.

@K890 I used the config bellow:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: infinispan-operator
  namespace: core
spec:
  replicas: 1
  selector:
    matchLabels:
      name: infinispan-operator
  template:
    metadata:
      labels:
        name: infinispan-operator
    spec:
      serviceAccountName: infinispan-operator
      containers:
      - name: infinispan-operator
        image: jboss/infinispan-operator:1.1.1
        ports:
        - containerPort: 60000
          name: metrics
        command:
        - infinispan-operator
        imagePullPolicy: Always
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: "infinispan-operator"
        - name: MAKE_DATADIR_WRITABLE
          value: "true"

mxandeco avatar Mar 19 '21 11:03 mxandeco

@mxandeco If I try to change the infinispan-operator spec directly by editing it, it is overridden in some time by the OLM/CSV. However, I was able to change this spec permanently by editing the csv resource infinispan-operator.v2.1.2 in the operators namespace and adding the MAKE_DATADIR_WRITABLE conf there.

P.S. If somebody gets an creationTimestamp= null error while updating the csv resource, copy-paste the other creationTimestamp which is not null to the place where it is null.

K890 avatar Mar 22 '21 09:03 K890

Unfortunately that does not work for me.

dicolasi avatar May 11 '21 13:05 dicolasi

@dicolasi what is your target operator deployment platform and the operator version?

dmvolod avatar May 11 '21 13:05 dmvolod 


2021-05-11 14:19:13,319 WARN  [io.qua.qut.run.ContentTypes] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) config-generator 2.1.3.Final native (powered by Quarkus 1.7.6.Final) started in 0.007s. 
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) Profile prod activated. 
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) Installed features: [cdi, qute]
2021-05-11 14:19:13,329 INFO  [io.quarkus] (main) config-generator stopped in 0.001s
14:19:18,099 INFO  (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.10+9-LTS
14:19:18,108 INFO  (main) [BOOT] JVM arguments = [-server, -Xmx200M, -Xms200M, -XX:MaxRAM=420M, -Dsun.zip.disableMemoryMapping=true, -XX:+UseSerialGC, -XX:MinHeapFreeRatio=5, -XX:MaxHeapFreeRatio=10, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.0.2.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap]
14:19:18,108 INFO  (main) [BOOT] PID = 97
14:19:18,215 INFO  (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
14:19:18,215 INFO  (main) [org.infinispan.SERVER] ISPN080017: Server configuration: infinispan.xml
14:19:18,216 INFO  (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
14:19:21,401 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
14:19:21,403 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
14:19:21,406 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory'
14:19:21,411 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
14:19:23,713 WARN  (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal
14:19:23,901 INFO  (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
14:19:26,705 INFO  (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.query.core.stats.impl.PersistenceContextInitializerImpl
14:19:26,707 INFO  (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Lockdown' 12.0.2.Final
14:19:26,709 WARN  (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
14:19:26,714 INFO  (main) [org.infinispan.server.core.RequestTracer] OpenTracing integration is disabled
14:19:26,715 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:158)
    at org.infinispan.server.tool.Main.run(Main.java:98)
    at org.infinispan.server.Bootstrap.main(Bootstrap.java:46)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.infinispan.server.loader.Loader.run(Loader.java:106)
    at org.infinispan.server.loader.Loader.main(Loader.java:51)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:751)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:717)
    at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:67)
    at org.infinispan.security.Security.doPrivileged(Security.java:45)
    at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
    at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:70)
    at org.infinispan.server.Server.run(Server.java:351)
    ... 9 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:87)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:64)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:341)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:237)
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:746)
    ... 15 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:81)
    ... 26 more
14:19:26,795 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
14:19:26,800 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped
stream closed

dicolasi avatar May 11 '21 14:05 dicolasi

@dmvolod

infinispan-operator.v2.1.3

dicolasi avatar May 11 '21 14:05 dicolasi

:pods Screenshot 2021-05-11 at 15 24 07

dicolasi avatar May 11 '21 14:05 dicolasi

@dicolasi where you've configured MAKE_DATADIR_WRITABLE env variable to configure this option?

dmvolod avatar May 11 '21 14:05 dmvolod

@dmvolod in the csvs.

dicolasi avatar May 11 '21 14:05 dicolasi

in the csvs.

Can you validate, that's applied to the operator deployment and Pod

dmvolod avatar May 11 '21 14:05 dmvolod

Interesting:

#
# clusterserviceversions.operators.coreos.com "infinispan-operator.v2.1.3" was not valid:
# * <nil>: Invalid value: "The edited file failed validation": [yaml: line 359: found a tab character that violates indentation, invalid character 'a' looking for beginning of value]

here the content of that bit:

          spec:
              containers:
              - command:
                - infinispan-operator
                env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['olm.targetNamespaces']
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: OPERATOR_NAME
                  value: infinispan-operator
                - name: RELATED_IMAGE_OPENJDK
                  value: quay.io/infinispan/server:12.0.2.Final
                - name: MAKE_DATADIR_WRITABLE
                  value: "true"

dicolasi avatar May 11 '21 14:05 dicolasi

  • : Invalid value: "The edited file failed validation": [yaml: line 359: found a tab character that violates indentation, invalid character 'a' looking for beginning of value]

That's the reason why OLM couldn't apply configuration to the Deployment. Please have a look at the operator deployment and Pod also

dmvolod avatar May 11 '21 14:05 dmvolod

what should I look for? The env variable is not there.

dicolasi avatar May 11 '21 14:05 dicolasi

what should I look for? The env variable is not there.

kubectl get deployment infinispan-operator -o yaml -n <namespace>

dmvolod avatar May 11 '21 14:05 dmvolod

@dmvolod just to clarify: is this a bug with the operator?

dicolasi avatar May 11 '21 14:05 dicolasi

just to clarify: is this a bug with the operator?

Not sure, this is specific k8s platform limitation and should work

dmvolod avatar May 11 '21 14:05 dmvolod

@dmvolod I am re-deploying the operator. Need 3 minutes.

dicolasi avatar May 11 '21 14:05 dicolasi 

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2021-05-11T14:53:39Z"
  generation: 2
  labels:
    olm.deployment-spec-hash: 7b99645b7f
    olm.owner: infinispan-operator.v2.1.3
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: cache
    operators.coreos.com/infinispan.cache: ""
  name: infinispan-operator
  namespace: cache
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: infinispan-operator.v2.1.3
    uid: 530740bb-b151-4fb4-9239-7e0e6ade486d
  resourceVersion: "95761466"
  selfLink: /apis/apps/v1/namespaces/cache/deployments/infinispan-operator
  uid: 121843e1-633e-4c7d-a767-1c15c716c372
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      name: infinispan-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        alm-examples: |-
          [
            {
              "apiVersion": "infinispan.org/v1",
              "kind": "Infinispan",
              "metadata": {
                "name": "example-infinispan"
              },
              "spec": {
                "replicas": 1
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Backup",
              "metadata": {
                "name": "example-backup"
              },
              "spec": {
                "cluster": "example-infinispan",
                "container": {
                  "cpu": "1000m",
                  "extraJvmOpts": "-Djava.property=me",
                  "memory": "1Gi"
                },
                "path": "asdasd"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Cache",
              "metadata": {
                "name": "example-cache"
              },
              "spec": {
                "adminAuth": {
                  "secretName": "basic-auth"
                },
                "clusterName": "example-infinispan",
                "name": "mycache"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Restore",
              "metadata": {
                "name": "example-restore"
              },
              "spec": {
                "cluster": "example-infinispan",
                "container": {
                  "cpu": "1000m",
                  "extraJvmOpts": "-Djava.property=me",
                  "memory": "1Gi"
                },
                "path": "asdasd"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Batch",
              "metadata": {
                "name": "example-batch",
              },
              "spec": {
                "cluster": "example-infinispan",
                "config": "create cache --template=org.infinispan.DIST_SYNC batch-cache"
              }
            },
          ]
        capabilities: Full Lifecycle
        categories: Database
        certified: "false"
        containerImage: quay.io/infinispan/operator:2.1.2.Final
        createdAt: "2021-03-18T09:00:00Z"
        description: Create and manage Infinispan clusters.
        olm.operatorGroup: operatorgroup
        olm.operatorNamespace: cache
        olm.targetNamespaces: cache
        operatorframework.io/properties: '{"properties":[{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Backup","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Batch","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Cache","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Infinispan","version":"v1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Restore","version":"v2alpha1"}},{"type":"olm.package","value":{"packageName":"infinispan","version":"2.1.3"}}]}'
        repository: https://github.com/infinispan/infinispan-operator
        support: Infinispan
      creationTimestamp: "2021-03-18T09:00:00Z"
      labels:
        name: infinispan-operator
    spec:
      containers:
      - command:
        - infinispan-operator
        env:
        - name: MAKE_DATADIR_WRITABLE
          value: "true"
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations['olm.targetNamespaces']
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: infinispan-operator
        - name: RELATED_IMAGE_OPENJDK
          value: quay.io/infinispan/server:12.0.2.Final
        - name: OPERATOR_CONDITION_NAME
          value: infinispan-operator.v2.1.3
        image: quay.io/infinispan/operator:2.1.2.Final
        imagePullPolicy: Always
        name: infinispan-operator
        ports:
        - containerPort: 60000
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: infinispan-operator
      serviceAccountName: infinispan-operator
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-05-11T14:53:43Z"
    lastUpdateTime: "2021-05-11T14:53:43Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-05-11T14:53:40Z"
    lastUpdateTime: "2021-05-11T14:56:22Z"
    message: ReplicaSet "infinispan-operator-5d695dfc55" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 2
  unavailableReplicas: 1
  updatedReplicas: 1

dicolasi avatar May 11 '21 14:05 dicolasi

I have modified the csv and now deploy has the environment variable set. No success however.

dicolasi avatar May 11 '21 14:05 dicolasi

Screenshot 2021-05-11 at 15 57 33

dicolasi avatar May 11 '21 14:05 dicolasi

ok sorted! The trick was to remove the old stateful set.

dicolasi avatar May 11 '21 14:05 dicolasi

Thanks for the support @dmvolod

dicolasi avatar May 11 '21 15:05 dicolasi

Thanks for the support

You are welcome

dmvolod avatar May 11 '21 15:05 dmvolod

@dmvolod still see the same error with a new installation:


2021-10-21 07:57:34,163 WARN  [io.qua.qut.run.ContentTypes[] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream
2021-10-21 07:57:34,163 INFO  [io.quarkus[] (main) config-generator 2.1.8.Final native (powered by Quarkus 1.7.6.Final) started in 0.075s. 
2021-10-21 07:57:34,164 INFO  [io.quarkus[] (main) Profile prod activated. 
2021-10-21 07:57:34,164 INFO  [io.quarkus[] (main) Installed features: [cdi, qute[]
2021-10-21 07:57:34,173 INFO  [io.quarkus[] (main) config-generator stopped in 0.000s
07:57:48,171 INFO  (main) [BOOT[] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.11+9-LTS
07:57:48,277 INFO  (main) [BOOT[] JVM arguments = [-server, -Dinfinispan.bind.address=0.0.0.0, -Xmx256m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.1.7.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap]
07:57:48,367 INFO  (main) [BOOT[] PID = 85
07:58:02,768 WARN  (main) [org.infinispan.PERSISTENCE[] ISPN000554: jboss-marshalling is deprecated and planned for removal
07:58:14,470 WARN  (main) [org.infinispan.CONTAINER[] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
07:58:14,567 FATAL (main) [org.infinispan.SERVER[] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:160)
    at org.infinispan.server.tool.Main.run(Main.java:98)
    at org.infinispan.server.Bootstrap.main(Bootstrap.java:47)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.infinispan.server.loader.Loader.run(Loader.java:106)
    at org.infinispan.server.loader.Loader.main(Loader.java:51)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:755)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:718)
    at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:67)
    at org.infinispan.security.Security.doPrivileged(Security.java:56)
    at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
    at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:70)
    at org.infinispan.server.Server.run(Server.java:360)
    ... 9 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:88)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:65)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:750)
    ... 15 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:82)
    ... 26 more
    ```

dicolasi avatar Oct 21 '21 07:10 dicolasi