kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] pulsar cluster hscale broker pod crash, upgrade KB from 0.7.5 to 0.8.3
Describe the bug A clear and concise description of what the bug is.
To Reproduce Steps to reproduce the behavior:
- instal KB 0.7.5
- create pulsar cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: pulsar-upkb781
namespace: default
spec:
affinity:
podAntiAffinity: Preferred
tenancy: SharedNode
topologyKeys:
- kubernetes.io/hostname
clusterDefinitionRef: pulsar
clusterVersionRef: pulsar-2.11.2
componentSpecs:
- classDefRef:
class: ""
componentDefRef: pulsar-broker
monitor: true
name: pulsar-broker
noCreatePDB: false
replicas: 1
resources:
limits:
cpu: 600m
memory: 644245094400m
requests:
cpu: 600m
memory: 644245094400m
rsmTransformPolicy: ToSts
- componentDefRef: bookies
monitor: true
name: bookies
noCreatePDB: false
replicas: 3
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 512Mi
rsmTransformPolicy: ToSts
volumeClaimTemplates:
- name: ledgers
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- name: journal
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- componentDefRef: zookeeper
monitor: true
name: zookeeper
noCreatePDB: false
replicas: 3
resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
rsmTransformPolicy: ToSts
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- componentDefRef: pulsar-proxy
monitor: true
name: pulsar-proxy
noCreatePDB: false
replicas: 1
resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
rsmTransformPolicy: ToSts
monitor: {}
resources:
cpu: "0"
memory: "0"
storage:
size: "0"
terminationPolicy: WipeOut
- hscale out broker 1 --> 2 success
kbcli_0.7.5 cluster hscale pulsar-upkb781 --components pulsar-broker --replicas 2 --auto-approve
- upgrade KB from 0.7.5 to 0.8.3
- hscale out broker 1 --> 2 failure
kbcli cluster hscale pulsar-upkb781 --components pulsar-broker --replicas 3 --auto-approve
- see error
➜ ~ kubectl get cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
pulsar-upkb781 pulsar pulsar-2.11.2 WipeOut Updating 74m
➜ ~
➜ ~ kubectl get pod
NAME READY STATUS RESTARTS AGE
kb-addon-csi-hostpath-driver-0 8/8 Running 0 82m
kb-addon-kubebench-bb5979455-64r25 1/1 Running 0 43m
kb-addon-minio-79f868bbf9-zzjbr 1/1 Running 0 82m
kb-addon-snapshot-controller-8484bbd44c-k7jvx 1/1 Running 0 83m
kubeblocks-6d4987bff6-4nl8s 1/1 Running 0 50m
kubeblocks-dataprotection-fb6dc4894-bsx85 1/1 Running 0 50m
pulsar-upkb781-bookies-0 2/2 Running 0 9m36s
pulsar-upkb781-bookies-1 2/2 Running 0 10m
pulsar-upkb781-bookies-2 2/2 Running 0 10m
pulsar-upkb781-pulsar-broker-0 3/3 Running 0 58m
pulsar-upkb781-pulsar-broker-1 3/3 Running 0 60m
pulsar-upkb781-pulsar-broker-2 2/3 CrashLoopBackOff 6 (4m33s ago) 10m
pulsar-upkb781-pulsar-proxy-0 2/2 Running 0 60m
pulsar-upkb781-zookeeper-0 2/2 Running 0 60m
pulsar-upkb781-zookeeper-1 2/2 Running 0 60m
pulsar-upkb781-zookeeper-2 2/2 Running 0 60m
➜ ~
➜ ~ kubectl get ops
NAME TYPE CLUSTER STATUS PROGRESS AGE
pulsar-upkb781-horizontalscaling-tncc8 HorizontalScaling pulsar-upkb781 Running 0/1 11m
describe pod
kubectl describe pod pulsar-upkb781-pulsar-broker-2
Name: pulsar-upkb781-pulsar-broker-2
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Mon, 15 Apr 2024 11:23:08 +0800
Labels: app.kubernetes.io/component=pulsar-broker
app.kubernetes.io/instance=pulsar-upkb781
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=pulsar
app.kubernetes.io/version=pulsar-2.11.2
apps.kubeblocks.io/cluster-uid=398b3dda-5d92-4970-8a8f-ac89ca1da2ab
apps.kubeblocks.io/component-name=pulsar-broker
apps.kubeblocks.io/workload-type=Stateful
clusterdefinition.kubeblocks.io/name=pulsar
clusterversion.kubeblocks.io/name=pulsar-2.11.2
controller-revision-hash=pulsar-upkb781-pulsar-broker-f4689f9dc
statefulset.kubernetes.io/pod-name=pulsar-upkb781-pulsar-broker-2
Annotations: apps.kubeblocks.io/component-replicas: 3
config.kubeblocks.io/restart-broker-config: fdd4688
kubeblocks.io/restart: 2024-04-15T02:33:18Z
Status: Running
IP: 10.244.18.26
IPs:
IP: 10.244.18.26
Controlled By: StatefulSet/pulsar-upkb781-pulsar-broker
Init Containers:
init-broker-cluster:
Container ID: docker://4343a0249b1cb2948ab9c3607e2fc176568835a0ca858da362b6032cf9e589da
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
Port: <none>
Host Port: <none>
Command:
/kb-scripts/init-broker.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 15 Apr 2024 11:23:09 +0800
Finished: Mon, 15 Apr 2024 11:23:09 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-broker-env-envfrom ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
brokerServiceUrl: pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
clusterName: $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
webServiceUrl: http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
Mounts:
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
init-sysctl:
Container ID: docker://921f20da52120a5d491f7947d077169883475db3684cd526471767dcdd2ffe20
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
Port: <none>
Host Port: <none>
Command:
/kb-scripts/init-broker-sysctl.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 15 Apr 2024 11:23:10 +0800
Finished: Mon, 15 Apr 2024 11:23:10 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
Mounts:
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
init-pulsar-tools:
Container ID: docker://d8fb905a422daf699cdd16783dd736bf7ab02423de9b7a35ff067c355799ceb3
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools:2.11
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools@sha256:ca94a8331febfe3a51b79ffe3c366a554e708c9a607fb9f6c0539cc28914a49a
Port: <none>
Host Port: <none>
Command:
cp
/bin/pulsar-admin
/kb/tools/pulsar-admin
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 15 Apr 2024 11:23:11 +0800
Finished: Mon, 15 Apr 2024 11:23:11 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
TOOLS_SCRIPTS_PATH: /opt/kb-tools/reload/broker-config
Mounts:
/kb/tools from kb-tools (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
/opt/pulsar/conf from pulsar-config (rw)
/opt/pulsar/none from broker-env (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
init-pulsar-client-config:
Container ID: docker://bcdc71eb4df5beab34d2dbccc3007c2e038f6205e1fc383ca661f901a28d6281
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
Port: <none>
Host Port: <none>
Command:
$(TOOLS_SCRIPTS_PATH)/install-pulsar-client-config.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 15 Apr 2024 11:23:12 +0800
Finished: Mon, 15 Apr 2024 11:23:12 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-broker-env-envfrom ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
TOOLS_SCRIPTS_PATH: /opt/kb-tools/reload/broker-config
Mounts:
/kb/tools from kb-tools (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
/opt/pulsar/conf from pulsar-config (rw)
/opt/pulsar/none from broker-env (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
Containers:
broker:
Container ID: docker://ce24b1766308865fd8a4f301caead39d809676dea1d037929e03e48bf5cc92c5
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:d394d32706c927c54b28a2e0f051ac7ee8ac25bdb69a3fb23d7de6510e4daa1a
Ports: 6650/TCP, 8080/TCP, 9092/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
sh
-c
Args:
set -x
/kb-scripts/merge_pulsar_config.py conf/client.conf /opt/pulsar/conf/client.conf && \
/kb-scripts/merge_pulsar_config.py conf/broker.conf /opt/pulsar/conf/broker.conf && \
bin/apply-config-from-env.py conf/broker.conf && \
bin/apply-config-from-env.py conf/client.conf && \
echo 'OK' > status;exec bin/pulsar broker
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 15 Apr 2024 11:29:30 +0800
Finished: Mon, 15 Apr 2024 11:29:34 +0800
Ready: False
Restart Count: 6
Limits:
cpu: 600m
memory: 644245094400m
Requests:
cpu: 600m
memory: 644245094400m
Liveness: http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
Readiness: http-get http://:http/status.html delay=30s timeout=100s period=10s #success=1 #failure=3
Startup: http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-broker-env-envfrom ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
SERVICE_PORT: 8080
POD_NAME: $(KB_POD_NAME)
PULSAR_PREFIX_brokerServicePort: 6650
PULSAR_PREFIX_internalListenerName: cluster
PULSAR_PREFIX_advertisedListeners: cluster:pulsar://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:6650
PULSAR_PREFIX_allowAutoTopicCreationType: partitioned
PULSAR_PREFIX_kafkaListeners: PLAINTEXT://0.0.0.0:9092
PULSAR_PREFIX_kafkaAdvertisedListeners: PLAINTEXT://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:9092
brokerServiceUrl: pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
clusterName: $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
webServiceUrl: http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
Mounts:
/kb-scripts from scripts (rw)
/kb/tools from kb-tools (rw)
/opt/pulsar/conf from pulsar-config (rw)
/opt/pulsar/none from broker-env (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
metrics:
Container ID: docker://4394fadfcfed2759cb30e35bc0ec8762011db5e51ea12b59476d8e03985173f4
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:a85d7d80d33de188ab39880d8d6013b663ebade9186371ce09a06c29dfa443ac
Port: 1234/TCP
Host Port: 0/TCP
Command:
/bin/agamotto
--config=/opt/agamotto/agamotto-config.yaml
--feature-gates=-pkg.translator.prometheus.NormalizeName
State: Running
Started: Mon, 15 Apr 2024 11:23:14 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
JOB_NAME: broker
SERVICE_PORT: 8080
POD_NAME: $(KB_POD_NAME)
POD_NAMESPACE: $(KB_NAMESPACE)
LOG_LEVEL: info
Mounts:
/opt/agamotto from agamotto-configuration (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
config-manager:
Container ID: docker://7196ad355b7140b0b90d33661799896b90490e733969cc60c09fa8bbad4258ce
Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.8.3-beta.7
Image ID: docker-pullable://infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:d5c966a4f96c5e60f7530fb08c55a40d30e501ca05e41e48574be3e4ab7ec220
Port: <none>
Host Port: <none>
Command:
env
Args:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
/bin/reloader
--log-level
info
--operator-update-enable
--tcp
9901
--config
/opt/config-manager/config-manager.yaml
State: Running
Started: Mon, 15 Apr 2024 11:23:14 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
pulsar-upkb781-pulsar-broker-env ConfigMap Optional: false
pulsar-upkb781-pulsar-broker-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: pulsar-upkb781-pulsar-broker-2 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: pulsar-upkb781
KB_COMP_NAME: pulsar-broker
KB_CLUSTER_COMP_NAME: pulsar-upkb781-pulsar-broker
KB_CLUSTER_UID_POSTFIX_8: ca1da2ab
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: pulsar-broker
TOOLS_PATH: /opt/kb-tools/reload/broker-config:/opt/config-manager:/kb/tools
Mounts:
/kb/tools from kb-tools (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
/opt/pulsar/conf from pulsar-config (rw)
/opt/pulsar/none from broker-env (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kc8n9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
agamotto-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pulsar-upkb781-pulsar-broker-agamotto-configuration
Optional: false
pulsar-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pulsar-upkb781-pulsar-broker-broker-config
Optional: false
broker-env:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pulsar-upkb781-pulsar-broker-broker-env
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pulsar-upkb781-pulsar-broker-pulsar-scripts
Optional: false
cm-script-broker-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-pulsar-tools-script-pulsar-upkb781
Optional: false
config-manager-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-pulsar-upkb781-pulsar-broker-config-manager-config
Optional: false
kb-tools:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-kc8n9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned default/pulsar-upkb781-pulsar-broker-2 to minikube
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
Normal Created 11m kubelet Created container init-broker-cluster
Normal Started 11m kubelet Started container init-broker-cluster
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
Normal Created 11m kubelet Created container init-sysctl
Normal Started 11m kubelet Started container init-sysctl
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools:2.11" already present on machine
Normal Created 11m kubelet Created container init-pulsar-tools
Normal Started 11m kubelet Started container init-pulsar-tools
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
Normal Created 11m kubelet Created container init-pulsar-client-config
Normal Started 11m kubelet Started container init-pulsar-client-config
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2" already present on machine
Normal Created 11m kubelet Created container metrics
Normal Pulled 11m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.8.3-beta.7" already present on machine
Normal Created 11m kubelet Created container config-manager
Normal Started 11m kubelet Started container config-manager
Normal Started 11m kubelet Started container metrics
Normal Pulled 11m (x2 over 11m) kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2" already present on machine
Normal Created 11m (x2 over 11m) kubelet Created container broker
Normal Started 11m (x2 over 11m) kubelet Started container broker
Warning Unhealthy 11m kubelet Startup probe failed: Get "http://10.244.18.26:8080/status.html": dial tcp 10.244.18.26:8080: connect: connection refused
Warning BackOff 71s (x56 over 11m) kubelet Back-off restarting failed container broker in pod pulsar-upkb781-pulsar-broker-2_default(9357606a-6f6b-4d01-b39b-b0bf48d7403b)
logs pod
➜ ~ kubectl logs pulsar-upkb781-pulsar-broker-2 broker
+ /kb-scripts/merge_pulsar_config.py conf/client.conf /opt/pulsar/conf/client.conf
Traceback (most recent call last):
File "/kb-scripts/merge_pulsar_config.py", line 59, in <module>
merge_config_files(dst_files, src_files)
File "/kb-scripts/merge_pulsar_config.py", line 45, in merge_config_files
lines, keys = load_config_file(src_file)
File "/kb-scripts/merge_pulsar_config.py", line 11, in load_config_file
for line in open(file):
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pulsar/conf/client.conf'
+ exec bin/pulsar broker
VM settings:
Max. Heap Size (Estimated): 186.00M
Using VM: OpenJDK 64-Bit Server VM
2024-04-15T03:34:50,040Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5bc2b487] error Uncaught exception in thread main: Required clusterName is null
java.lang.IllegalArgumentException: Required clusterName is null
at org.apache.pulsar.common.configuration.PulsarConfigurationLoader.isComplete(PulsarConfigurationLoader.java:153)
at org.apache.pulsar.PulsarBrokerStarter.loadConfig(PulsarBrokerStarter.java:72)
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.<init>(PulsarBrokerStarter.java:169)
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331)
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
This issue has been marked as stale because it has been open for 30 days with no activity
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}