kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] doris cluster component fe create failed on GKE
Describe the bug Error: failed to generate container "6ed13e4ae34749b117f8ddd25923ff47f09139f56ec68306976651d8014b3b5d" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
To Reproduce Steps to reproduce the behavior:
- create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: doris-cluster
namespace: default
spec:
clusterDefinitionRef: doris
clusterVersionRef: doris-2.0.3
terminationPolicy: Delete
componentSpecs:
- name: doris-fe
componentDefRef: doris-fe
replicas: 3
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- componentDefRef: doris-be
name: doris-be
replicas: 3
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- componentDefRef: doris-cn
name: doris-cn
replicas: 3
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
- See error
kubectl get pod
NAME READY STATUS RESTARTS AGE
doris-cluster-doris-be-0 1/1 Running 0 3m44s
doris-cluster-doris-be-1 1/1 Running 0 3m44s
doris-cluster-doris-be-2 1/1 Running 0 3m44s
doris-cluster-doris-cn-0 1/1 Running 0 3m46s
doris-cluster-doris-cn-1 1/1 Running 0 3m46s
doris-cluster-doris-cn-2 1/1 Running 0 3m46s
doris-cluster-doris-fe-0 2/3 CreateContainerError 0 3m46s
doris-cluster-doris-fe-1 2/3 CreateContainerError 0 3m46s
doris-cluster-doris-fe-2 2/3 CreateContainerError 0 3m46s
describe pod
kubectl describe pod doris-cluster-doris-fe-0
Name: doris-cluster-doris-fe-0
Namespace: default
Priority: 0
Node: gke-infracreate-gke-default-pool-3154c215-ht4s/10.10.0.66
Start Time: Thu, 01 Feb 2024 18:50:38 +0800
Labels: app.kubernetes.io/component=doris-fe
app.kubernetes.io/instance=doris-cluster
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=doris
app.kubernetes.io/version=
apps.kubeblocks.io/component-name=doris-fe
controller-revision-hash=doris-cluster-doris-fe-5dfcccc76b
statefulset.kubernetes.io/pod-name=doris-cluster-doris-fe-0
Annotations: apps.kubeblocks.io/component-replicas: 3
Status: Pending
IP: 10.128.4.9
IPs:
IP: 10.128.4.9
Controlled By: StatefulSet/doris-cluster-doris-fe
Containers:
doris-fe:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe
Image ID:
Ports: 8030/TCP, 9020/TCP, 9030/TCP, 9010/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 512Mi
Requests:
cpu: 100m
memory: 512Mi
Environment Variables from:
doris-cluster-doris-fe-env ConfigMap Optional: false
doris-cluster-doris-fe-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: doris-cluster-doris-fe-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
APP_NAMESPACE: $(KB_NAMESPACE)
BUILD_TYPE: k8s
FE_INIT_NUMBER: 3
CN_SERVICE: $(KB_CLUSTER_NAME)-doris-cn-headless
CN_STATEFULSET: $(KB_CLUSTER_NAME)-doris-cn
BE_SERVICE: $(KB_CLUSTER_NAME)-doris-be-headless
BE_STATEFULSET: $(KB_CLUSTER_NAME)-doris-be
FE_SERVICE: $(KB_CLUSTER_NAME)-doris-fe-headless
FE_STATEFULSET: $(KB_CLUSTER_NAME)-doris-fe
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'doris-cluster-conn-credential'> Optional: false
Mounts:
/opt/apache-doris/fe/bin/mysql_root.sh from update-root (rw,path="mysql_root.sh")
/opt/apache-doris/fe/conf from conf (rw)
/opt/apache-doris/fe/doris-meta from data (rw)
/root/.kube/config from kube (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
mysql-root-script:
Container ID: containerd://1c688f80f9ce3db049be5ce6ba5719ad1830636c28cd990a19c8938addceae23
Image: registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe
Image ID: registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris@sha256:65618f340a647eb0d781e7a4052992a42d52b07d0e15c2143aed7a25880da624
Port: <none>
Host Port: <none>
Command:
sh
-c
/opt/apache-doris/fe/bin/mysql_root.sh > /opt/apache-doris/fe/bin/update_root.log 2>&1 &
while true; do sleep 1; done
State: Running
Started: Thu, 01 Feb 2024 18:50:48 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 2G
Requests:
cpu: 500m
memory: 1G
Environment Variables from:
doris-cluster-doris-fe-env ConfigMap Optional: false
doris-cluster-doris-fe-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: doris-cluster-doris-fe-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'doris-cluster-conn-credential'> Optional: false
Mounts:
/opt/apache-doris/fe/bin/mysql_root.sh from update-root (rw,path="mysql_root.sh")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
config-manager:
Container ID: containerd://99c3bdc3da5939684708289bc2b1384a588591e1e25212b16d19cbb9faf2d29d
Image: docker.io/apecloud/kubeblocks-tools:0.8.2-beta.4
Image ID: docker.io/apecloud/kubeblocks-tools@sha256:d4ced3a9c566b41b8df8c3eba89a3f8c5625fd6a6d8095561b13562f49a34b40
Port: <none>
Host Port: <none>
Command:
env
Args:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
/bin/reloader
--log-level
info
--operator-update-enable
--tcp
9901
--config
/opt/config-manager/config-manager.yaml
State: Running
Started: Thu, 01 Feb 2024 18:50:48 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
doris-cluster-doris-fe-env ConfigMap Optional: false
doris-cluster-doris-fe-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: doris-cluster-doris-fe-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: doris-fe
TOOLS_PATH: /opt/kb-tools/reload/doris-fe-conf-configuration:/opt/config-manager
Mounts:
/opt/apache-doris/fe/conf from conf (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/doris-fe-conf-configuration from cm-script-doris-fe-conf-configuration (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-doris-cluster-doris-fe-0
ReadOnly: false
kube:
Type: HostPath (bare host directory volume)
Path: /root/.kube/config
HostPathType:
conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: doris-cluster-doris-fe-doris-fe-conf-configuration
Optional: false
update-root:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: doris-cluster-doris-fe-doris-fe-root-script
Optional: false
cm-script-doris-fe-conf-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-doris-tools-script-doris-cluster
Optional: false
config-manager-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-doris-cluster-doris-fe-config-manager-config
Optional: false
kube-api-access-8bl6m:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m27s default-scheduler Successfully assigned default/doris-cluster-doris-fe-0 to gke-infracreate-gke-default-pool-3154c215-ht4s
Normal SuccessfulAttachVolume 4m20s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-40d97094-0e6d-4f82-89b5-b7eb526481c7"
Warning Failed 4m18s kubelet Error: failed to generate container "1b7fd46e44134a89e944c1859882e96db772d75ffaf14d55c134762e67c47f54" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Normal Pulled 4m18s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe" already present on machine
Normal Created 4m17s kubelet Created container config-manager
Normal Created 4m17s kubelet Created container mysql-root-script
Normal Started 4m17s kubelet Started container mysql-root-script
Normal Pulled 4m17s kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.8.2-beta.4" already present on machine
Normal Started 4m17s kubelet Started container config-manager
Warning Failed 4m16s kubelet Error: failed to generate container "6f841e1c059030b532e8af4b61e4833226a95206faf365bc50fbec7551934275" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 4m15s kubelet Error: failed to generate container "5fe445da971ca1f9084b474746624e70063035253764a8473d17f85afca1978b" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 4m3s kubelet Error: failed to generate container "e4871b1c58f89d657f57f145df1a0580509424657cf5ee64610c39a9d62907ea" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 3m50s kubelet Error: failed to generate container "7cdab9cfa2587df08889fb1440374da0ae7d9341b43b4c5090173c15af730400" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 3m37s kubelet Error: failed to generate container "ea16e5febc287d27f44a343b4cc900e998c9f3509cd0bde65cc5dc5d25c0c087" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 3m24s kubelet Error: failed to generate container "dc76c483b6ed365202eaa8d83c10e2763fa88b10bacd2c95eefa8a03306c53c4" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 3m9s kubelet Error: failed to generate container "c61a34d9e508fee53a9caacb7745d490e48d996d8648ea8fc61ed1617be1c20c" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning Failed 2m58s kubelet Error: failed to generate container "6ed13e4ae34749b117f8ddd25923ff47f09139f56ec68306976651d8014b3b5d" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Normal Pulled 2m44s (x10 over 4m18s) kubelet Container image "registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe" already present on machine
➜ ~
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
