Elias Abacioglu

I just checked by deleteing the .status key space. ``` $ curl -sJL https://github.com/banzaicloud/koperator/releases/download/v0.20.0/kafka-operator.crds.yaml | yq e 'del(.status)' - | kubectl apply --server-side -f - customresourcedefinition.apiextensions.k8s.io/kafkaclusters.kafka.banzaicloud.io serverside-applied customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.banzaicloud.io serverside-applied customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.banzaicloud.io...

Ok, lets make an action point on this. - Make sure that `.status` is not part of the CRDs Also I wonder, it is possible to make helm use server-side...

Cluster is fine. Ok, it's weird, I cannot replicate this with other user accounts, but I still can't delete the KafkaUser `myuser`. It's stuck in removal.

I cannot run kafka-acls.sh ``` $ kubectl exec -it kafka-0-j7qhs -c kafka -- kafka-acls.sh --bootstrap-server 0.0.0.0:29092 --list Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown...

I also ran a ``` /usr/local/openjdk-11/bin/java -Xmx2G -Xms2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/opt/kafka/bin/../logs -Dlog4j.configuration=file:/opt/kafka/bin/../config/tools-log4j.properties -cp '/opt/kafka/libs/extensions/*:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.8.1.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.8.1.jar:/opt/kafka/bin/../libs/connect-file-2.8.1.jar:/opt/kafka/bin/../libs/connect-json-2.8.1.jar:/opt/kafka/bin/../libs/connect-mirror-2.8.1.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.8.1.jar:/opt/kafka/bin/../libs/connect-runtime-2.8.1.jar:/opt/kafka/bin/../libs/connect-transforms-2.8.1.jar:/opt/kafka/bin/../libs/extensions:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.5.jar:/opt/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.10.5.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.34.jar:/opt/kafka/bin/../libs/jersey-common-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka/bin/../libs/jersey-server-2.34.jar:/opt/kafka/bin/../libs/jetty-client-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-http-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-io-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-security-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-server-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jline-3.12.1.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka-clients-2.8.1.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.8.1.jar:/opt/kafka/bin/../libs/kafka-metadata-2.8.1.jar:/opt/kafka/bin/../libs/kafka-raft-2.8.1.jar:/opt/kafka/bin/../libs/kafka-shell-2.8.1.jar:/opt/kafka/bin/../libs/kafka-streams-2.8.1.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.8.1.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-2.8.1.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.8.1.jar:/opt/kafka/bin/../libs/kafka-tools-2.8.1.jar:/opt/kafka/bin/../libs/kafka_2.13-2.8.1-sources.jar:/opt/kafka/bin/../libs/kafka_2.13-2.8.1.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.62.Final.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.3.0.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/opt/kafka/bin/../libs/scala-library-2.13.5.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.5.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.1.jar:/opt/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.9-1.jar' kafka.admin.AclCommand --list --bootstrap-server 0.0.0.0:29092 ``` But the pod...

I've tried [kafka-pod-ip]:9094/29092/29093, I've tried kafka-headless:9094/29092/29093. They all result in this error ``` Error while executing ACL command: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeAcls, deadlineMs=1641821474583, tries=1, nextAllowedTryMs=1641821474684) timed out at 1641821474584 after 1 attempt(s)...

```yaml apiVersion: kafka.banzaicloud.io/v1beta1 kind: KafkaCluster metadata: name: kafka namespace: kafka-dev spec: headlessServiceEnabled: true zkAddresses: - "zookeeper-client:2181" zkPath: "/kafka" propagateLabels: false oneBrokerPerNode: false clusterImage: "ghcr.io/banzaicloud/kafka:2.13-2.8.1" readOnlyConfig: | auto.create.topics.enable=false default.replication.factor=3 cruise.control.metrics.topic.auto.create=true cruise.control.metrics.topic.num.partitions=1...

``` Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=my-topic, patternType=LITERAL)`: (principal=User:CN=myuser, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:CN=myuser, host=*, operation=WRITE, permissionType=ALLOW) (principal=User:CN=myuser, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW) (principal=User:CN=myuser, host=*, operation=CREATE, permissionType=ALLOW) (principal=User:CN=myuser, host=*, operation=READ, permissionType=ALLOW) Current ACLs...

Sure, here are the logs with verbose enabled ```json {"level":"info","ts":"2022-01-18T09:51:44.052Z","logger":"controllers.KafkaUser","msg":"Reconciling KafkaUser","kafkauser":"kafka-dev/myuser","Request.Name":"myuser"} {"level":"info","ts":"2022-01-18T09:51:44.064Z","logger":"controllers.KafkaUser","msg":"Kafka user is marked for deletion, revoking certificates","kafkauser":"kafka-dev/myuser","Request.Name":"myuser"} {"level":"info","ts":"2022-01-18T09:51:44.064Z","logger":"controllers.KafkaUser","msg":"Deleting user ACLs from kafka","kafkauser":"kafka-dev/myuser","Request.Name":"myuser"} {"level":"info","ts":"2022-01-18T09:51:44.773Z","logger":"kafka_util","msg":"Kafka client closed cleanly"} {"level":"info","ts":"2022-01-18T09:51:44.773Z","logger":"controllers.KafkaUser","msg":"failed to...

@stoader They are listed above. However I myself am not able to reproduce this. I simply let that user exist as this was a dev cluster. I mostly reported this...