ceph-helm
ceph-helm copied to clipboard
Published
20 hours ago •
ceph
ceph
[secrets "ceph-bootstrap-osd-keyring" not found] ceph-osd fail on k8s 1.10.1 install by kubeadm
kubectl get po -n ceph
NAME READY STATUS RESTARTS AGE
ceph-mds-696bd98bdb-bnvpg 0/1 Pending 0 18m
ceph-mds-keyring-generator-q679r 0/1 Completed 0 18m
ceph-mgr-6d5f86d9c4-nr76h 1/1 Running 1 18m
ceph-mgr-keyring-generator-v825z 0/1 Completed 0 18m
ceph-mon-86lth 1/1 Running 0 18m
ceph-mon-check-74d98c5b95-wf9tm 1/1 Running 0 18m
ceph-mon-keyring-generator-rfg8j 0/1 Completed 0 18m
ceph-mon-pp5hc 1/1 Running 0 18m
ceph-namespace-client-key-cleaner-g9dri-sjmqd 0/1 Completed 0 1h
ceph-namespace-client-key-cleaner-qwkee-pdkh6 0/1 Completed 0 21m
ceph-namespace-client-key-cleaner-t25ui-5gkb7 0/1 Completed 0 2d
ceph-namespace-client-key-generator-xk4w6 0/1 Completed 0 18m
ceph-osd-dev-sda-6jbgd 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-dev-sda-khfhw 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-dev-sda-krkjf 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-keyring-generator-mvktj 0/1 Completed 0 18m
ceph-rbd-provisioner-b58659dc9-nhx2q 1/1 Running 0 18m
ceph-rbd-provisioner-b58659dc9-nnlh2 1/1 Running 0 18m
ceph-rgw-5bd9dd66c5-gh946 0/1 Pending 0 18m
ceph-rgw-keyring-generator-dz9kd 0/1 Completed 0 18m
ceph-storage-admin-key-cleaner-1as0t-fq589 0/1 Completed 0 1h
ceph-storage-admin-key-cleaner-oayjp-fglzr 0/1 Completed 0 2d
ceph-storage-admin-key-cleaner-zemvx-jxn7c 0/1 Completed 0 21m
ceph-storage-keys-generator-szps9 0/1 Completed 0 18m
Version of Helm and Kubernetes:
- k8s 1.10.1
- helm
helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Which chart:
commit 70681c8218e75bb10acc7dc210b791b69545ce0d
Merge: a4fa8a1 6cb2e1e
Author: Huamin Chen <[email protected]>
Date: Thu Apr 26 12:11:43 2018 -0400
- pod describe
kubectl describe -n ceph po ceph-osd-dev-sda-6jbgd
Name: ceph-osd-dev-sda-6jbgd
Namespace: ceph
Node: k8s-2/192.168.16.40
Start Time: Sat, 28 Apr 2018 15:54:08 +0800
Labels: application=ceph
component=osd
controller-revision-hash=1450926272
pod-template-generation=1
release_group=ceph
Annotations: <none>
Status: Pending
IP: 192.168.16.40
Controlled By: DaemonSet/ceph-osd-dev-sda
Init Containers:
init:
Container ID: docker://8c9536cc4c5d811f57ba6349c87245121651841f52db682f858ae0ac70555856
Image: docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0
Image ID: docker-pullable://kolla/ubuntu-source-kubernetes-entrypoint@sha256:75116ab2f9f65c5fc078e68ce7facd66c1c57496947f37b7209b32f94925e53b
Port: <none>
Host Port: <none>
Command:
kubernetes-entrypoint
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 28 Apr 2018 15:54:34 +0800
Finished: Sat, 28 Apr 2018 15:54:36 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: ceph-osd-dev-sda-6jbgd (v1:metadata.name)
NAMESPACE: ceph (v1:metadata.namespace)
INTERFACE_NAME: eth0
DEPENDENCY_SERVICE: ceph-mon
DEPENDENCY_JOBS:
DEPENDENCY_DAEMONSET:
DEPENDENCY_CONTAINER:
COMMAND: echo done
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
ceph-init-dirs:
Container ID: docker://1562879ebbc52c47cfd9fb292339e548d26450207846ff6eeb38594569d5ec5f
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID: docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
Port: <none>
Host Port: <none>
Command:
/tmp/init_dirs.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 28 Apr 2018 15:54:38 +0800
Finished: Sat, 28 Apr 2018 15:54:39 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/run from pod-run (rw)
/tmp/init_dirs.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
osd-prepare-pod:
Container ID: docker://2b5bed33de8f35533eb72ef3208010153b904a8ed34c527a4916b88f549d5f6b
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID: docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
Port: <none>
Host Port: <none>
Command:
/start_osd.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 28 Apr 2018 16:11:06 +0800
Finished: Sat, 28 Apr 2018 16:11:07 +0800
Ready: False
Restart Count: 8
Environment:
CEPH_DAEMON: osd_ceph_disk_prepare
KV_TYPE: k8s
CLUSTER: ceph
CEPH_GET_ADMIN_KEY: 1
OSD_DEVICE: /dev/mapper/centos-root
HOSTNAME: (v1:spec.nodeName)
Mounts:
/common_functions.sh from ceph-bin (ro)
/dev from devices (rw)
/etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
/etc/ceph/ceph.conf from ceph-etc (ro)
/etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
/osd_activate_journal.sh from ceph-bin (ro)
/osd_disk_activate.sh from ceph-bin (ro)
/osd_disk_prepare.sh from ceph-bin (ro)
/osd_disks.sh from ceph-bin (ro)
/run from pod-run (rw)
/start_osd.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
/var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
/var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
Containers:
osd-activate-pod:
Container ID:
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID:
Port: <none>
Host Port: <none>
Command:
/start_osd.sh
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: tcp-socket :6800 delay=60s timeout=5s period=10s #success=1 #failure=3
Readiness: tcp-socket :6800 delay=0s timeout=5s period=10s #success=1 #failure=3
Environment:
CEPH_DAEMON: osd_ceph_disk_activate
KV_TYPE: k8s
CLUSTER: ceph
CEPH_GET_ADMIN_KEY: 1
OSD_DEVICE: /dev/mapper/centos-root
HOSTNAME: (v1:spec.nodeName)
Mounts:
/common_functions.sh from ceph-bin (ro)
/dev from devices (rw)
/etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
/etc/ceph/ceph.conf from ceph-etc (ro)
/etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
/osd_activate_journal.sh from ceph-bin (ro)
/osd_disk_activate.sh from ceph-bin (ro)
/osd_disk_prepare.sh from ceph-bin (ro)
/osd_disks.sh from ceph-bin (ro)
/run from pod-run (rw)
/start_osd.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
/var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
/var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
/var/log/ceph from pod-var-log-ceph (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
Conditions:
Type Status
Initialized False
Ready False
PodScheduled True
Volumes:
devices:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
pod-var-lib-ceph:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
pod-var-log-ceph:
Type: HostPath (bare host directory volume)
Path: /var/log/ceph/osd
HostPathType:
pod-run:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
ceph-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ceph-bin
Optional: false
ceph-etc:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ceph-etc
Optional: false
ceph-client-admin-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-client-admin-keyring
Optional: false
ceph-mon-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-mon-keyring
Optional: false
ceph-bootstrap-osd-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-osd-keyring
Optional: false
ceph-bootstrap-mds-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-mds-keyring
Optional: false
ceph-bootstrap-rgw-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-rgw-keyring
Optional: false
default-token-z5m75:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-z5m75
Optional: false
QoS Class: BestEffort
Node-Selectors: ceph-osd=enabled
ceph-osd-device-dev-sda=enabled
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-var-log-ceph"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "devices"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-var-lib-ceph"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-run"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-bin"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-etc"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "default-token-z5m75"
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-client-admin-keyring"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-bootstrap-osd-keyring"
Warning FailedMount 19m (x4 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 19m (x4 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Normal SuccessfulMountVolume 19m (x2 over 19m) kubelet, k8s-2 (combined from similar events): MountVolume.SetUp succeeded for volume "ceph-mon-keyring"
Warning BackOff 4m (x65 over 18m) kubelet, k8s-2 Back-off restarting failed container
- pod log
kubectl logs -n ceph ceph-osd-dev-sda-6jbgd
Error from server (BadRequest): container "osd-activate-pod" in pod "ceph-osd-dev-sda-6jbgd" is waiting to start: PodInitializing
- tiller log
[tiller] 2018/04/28 07:34:20 preparing install for ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[tiller] 2018/04/28 07:34:20 rendering ceph chart using values
2018/04/28 07:34:20 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:34:20 performing install for ceph
[tiller] 2018/04/28 07:34:20 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for pre-install ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[storage] 2018/04/28 07:34:20 creating release "ceph.v1"
[kube] 2018/04/28 07:34:20 building resources from manifest
[kube] 2018/04/28 07:34:20 creating 22 resource(s)
[tiller] 2018/04/28 07:34:20 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for post-install ceph
[storage] 2018/04/28 07:34:20 updating release "ceph.v1"
[storage] 2018/04/28 07:34:21 getting last revision of "ceph"
[storage] 2018/04/28 07:34:21 getting release history for "ceph"
[kube] 2018/04/28 07:34:21 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:34:21 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:34:21 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-storage-keys-generator
[storage] 2018/04/28 07:50:39 getting release history for "ceph"
[tiller] 2018/04/28 07:50:39 uninstall: Deleting ceph
[tiller] 2018/04/28 07:50:39 executing 2 pre-delete hooks for ceph
[kube] 2018/04/28 07:50:39 building resources from manifest
[kube] 2018/04/28 07:50:39 creating 1 resource(s)
[kube] 2018/04/28 07:50:39 Watching for changes to Job ceph-namespace-client-key-cleaner-qwkee with timeout of 5m0s
[kube] 2018/04/28 07:50:39 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: ADDED
[kube] 2018/04/28 07:50:39 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:40 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:40 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:42 building resources from manifest
[kube] 2018/04/28 07:50:42 creating 1 resource(s)
[kube] 2018/04/28 07:50:42 Watching for changes to Job ceph-storage-admin-key-cleaner-zemvx with timeout of 5m0s
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: ADDED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:45 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[tiller] 2018/04/28 07:50:45 hooks complete for pre-delete ceph
[storage] 2018/04/28 07:50:45 updating release "ceph.v1"
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mon" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-rgw" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mds-keyring-generator" Job
[kube] 2018/04/28 07:50:45 Using reaper for deleting "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:50:47 Starting delete for "ceph-mgr-keyring-generator" Job
[kube] 2018/04/28 07:50:47 Using reaper for deleting "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:50:49 Starting delete for "ceph-mon-keyring-generator" Job
[kube] 2018/04/28 07:50:49 Using reaper for deleting "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:50:52 Starting delete for "ceph-rgw-keyring-generator" Job
[kube] 2018/04/28 07:50:52 Using reaper for deleting "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:50:54 Starting delete for "ceph-osd-keyring-generator" Job
[kube] 2018/04/28 07:50:54 Using reaper for deleting "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:50:56 Starting delete for "ceph-namespace-client-key-generator" Job
[kube] 2018/04/28 07:50:56 Using reaper for deleting "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:50:58 Starting delete for "ceph-storage-keys-generator" Job
[kube] 2018/04/28 07:50:58 Using reaper for deleting "ceph-storage-keys-generator"
[kube] 2018/04/28 07:51:00 Starting delete for "ceph-mds" Deployment
[kube] 2018/04/28 07:51:00 Using reaper for deleting "ceph-mds"
[kube] 2018/04/28 07:51:03 Starting delete for "ceph-mgr" Deployment
[kube] 2018/04/28 07:51:03 Using reaper for deleting "ceph-mgr"
[kube] 2018/04/28 07:51:06 Starting delete for "ceph-mon-check" Deployment
[kube] 2018/04/28 07:51:06 Using reaper for deleting "ceph-mon-check"
[kube] 2018/04/28 07:51:09 Starting delete for "ceph-rbd-provisioner" Deployment
[kube] 2018/04/28 07:51:09 Using reaper for deleting "ceph-rbd-provisioner"
[kube] 2018/04/28 07:51:12 Starting delete for "ceph-rgw" Deployment
[kube] 2018/04/28 07:51:12 Using reaper for deleting "ceph-rgw"
[kube] 2018/04/28 07:51:15 Starting delete for "ceph-mon" DaemonSet
[kube] 2018/04/28 07:51:15 Using reaper for deleting "ceph-mon"
[kube] 2018/04/28 07:51:58 Starting delete for "ceph-osd-dev-sda" DaemonSet
[kube] 2018/04/28 07:51:58 Using reaper for deleting "ceph-osd-dev-sda"
[kube] 2018/04/28 07:52:58 Starting delete for "ceph-rbd" StorageClass
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin-clients" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-etc" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-templates" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-keystone-user-rgw" Secret
2018/04/28 07:52:59 uninstall: Failed deletion of "ceph": no objects visited
[tiller] 2018/04/28 07:52:59 error: object not found, skipping delete
[tiller] 2018/04/28 07:52:59 executing 2 post-delete hooks for ceph
[tiller] 2018/04/28 07:52:59 hooks complete for post-delete ceph
[tiller] 2018/04/28 07:52:59 purge requested for ceph
[storage] 2018/04/28 07:52:59 deleting release "ceph.v1"
[tiller] 2018/04/28 07:54:07 preparing install for ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[tiller] 2018/04/28 07:54:07 rendering ceph chart using values
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:54:07 performing install for ceph
[tiller] 2018/04/28 07:54:07 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:54:07 hooks complete for pre-install ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[storage] 2018/04/28 07:54:07 creating release "ceph.v1"
[kube] 2018/04/28 07:54:07 building resources from manifest
[kube] 2018/04/28 07:54:07 creating 22 resource(s)
[tiller] 2018/04/28 07:54:09 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:54:09 hooks complete for post-install ceph
[storage] 2018/04/28 07:54:09 updating release "ceph.v1"
[storage] 2018/04/28 07:54:09 getting last revision of "ceph"
[storage] 2018/04/28 07:54:09 getting release history for "ceph"
[kube] 2018/04/28 07:54:09 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:54:09 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:54:09 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-storage-keys-generator
system centos CentOS Linux release 7.4.1708 (Core)
install by vmware
sudo fdisk -l
[sudo] password for k8s:
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000aae00
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 104857599 51379200 8e Linux LVM
Disk /dev/mapper/centos-root: 50.5 GB, 50457477120 bytes, 98549760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Apr 19 03:44:34 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=fdf34de5-5810-4b0e-a7fe-4b92c03cf582 /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
- values config
network:
public: 192.168.0.0/16
cluster: 192.168.0.0/16
port:
mon: 6789
rgw: 8088
enable_zap_and_potentially_lose_data: false
osd_devices:
- name: dev-sda
device: /dev/mapper/centos-root
# journal: /dev/sdf
zap: "1"
ceph_mgr_enabled_modules:
- restful
- status
- dashboard
# You can configure your mgr modules
# below. Each module has its own set
# of key/value. Refer to the doc
# above for more info. For example:
ceph_mgr_modules_config:
dashboard:
port: 7000
localpool:
failure_domain: host
subtree: rack
pg_num: "128"
num_rep: "3"
min_size: "2"
storageclass:
provision_storage_class: true
provisioner: ceph.com/rbd
name: ceph-rbd
monitors: null
pool: rbd
admin_id: admin
admin_secret_name: pvc-ceph-conf-combined-storageclass
admin_secret_namespace: ceph
user_id: admin
user_secret_name: pvc-ceph-client-key
image_format: "2"
image_features: layering
I met that error before when I install it after uninstall a "failed helm install ceph before PR #54" . So I doubt that there maybe some files were not cleaned by the "helm delete --purge ". So I did a clean installation started from all new VMs and kubernetes cluster, finally got a successful installation of ceph.
BTW: It seems that you config the OSD device to use the OS device ??? I think you should add one or more bland disk for OSD device, NOT the OS root device
@whmzsu thank you. i deleted the /var/log/ceph /var/lib/ceph-helm also clean the k8s all source under ceph namespace.
@foxundermoon I try to install the ceph again after uninstall the success installation, delete the ceph-helm dir , and got the same error like your. So I think there must be something not cleared by the helm while uninstalling.
We also found this issue on Kubernetes 1.9.2. As we try all is clear all helm environment/reinital k8s farm and try more than 3 times still issue this
@whmzsu still issue. i am reinstall on other 3 servers.
still meet the error
Normal SuccessfulMountVolume 54m kubelet, bj-h03-11-cm.getui MountVolume.SetUp succeeded for volume "default-token-xp4fc"
Warning FailedMount 54m (x3 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Normal Pulled 48m kubelet, bj-h03-11-cm.getui Successfully pulled image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0"
I am getting same problem.Problem is "/etc/ceph/ceph.mon.keyring" I do not have such a path "/etc/ceph" or "var/lib/ceph". How can I all do this with a VM ? I do not want to run a host with minikube.
Installation (Kubernetes + Helm)
minikube version: v0.27.0
helm version Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
stat: can't stat '/var/log/ceph/ceph.audit.log': No such file or directory container "osd-activate-pod" in pod "ceph-osd-dev-sdd-pzqck" is waiting to start: PodInitializing MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found Back-off restarting failed container
{ "name": "ceph-mon-keyring", "readOnly": true, "mountPath": "/etc/ceph/ceph.mon.keyring", "subPath": "ceph.mon.keyring" },
Encountered with the same issue in discussion
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 1m (x3 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 1m (x4 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 1m (x5 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 1m (x6 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Warning FailedMount 1m (x6 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Normal Pulled 50s kubelet, server-6 Container image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0" already present on machine
When i tried for the first time everything worked fine. As soon as i deleted the cluster using helm delete --purge ceph --timeout 1000 --debug and retried helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml i started getting these errors.
Also i do not have directories like /etc/ceph /var/lib/ceph /var/log/ceph /var/lib/ceph-helm.
Need help to get find A) Clean way to delete the cluster B) Fix this issue.
@rootfs any suggestions ?
@foxundermoon @githubfoam did you ever managed to get this working. Whats the clean way to purge the cluster ?
You must remove the partitions on target devices before installing a new ceph cluster.
Hi All,
I have exactly the same issue with Kubernetes 1.11.2. Is there still no fix or workaround for this ? @sunx The osd discs/partitions are untouched as the osd Pod are not coming up. Cheers!
The OSD Pods still not coming up, a "kubectl describe pod xxxxxx" shows:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning BackOff 1m (x41 over 10m) kubelet, k8s-ceph3 Back-off restarting failed container
But all of these Secrets are existing in the same namespace.
same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0. osd-prepare-pod with mounting event exceptions like @xforze .
any ideas?
same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0. osd-prepare-pod with mounting event exceptions like @xforze .
Same issue occured on K8S v1.11.2 / Ubuntu 18.04. However after I deleted pods, they mounted secrets successfully.
After that I have stumbled upon other error related to reinstall. At fault was storage disks with old information from previous install. @mamoit incorrectly suggested to use fdisk, where you should use gdisk to wipe/recreate partitions! Cephs expects GPT partition table and fdisk does not properly handle that.
To verify, check logs from init container:
kubectl -n ceph logs ceph-osd-dev-sdc-XXXX -c osd-prepare-pod
I was getting the following error:
create_partition: Creating journal partition num 2 size 5120 on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!
Invalid partition data!
/sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc' failed with status code 2
After I have recreated GPT partition table (option 2) with gdisk /dev/sdc, pod successfully initialized.
cool @elvinasp ! I should give a try for that. I moved to rook.io project to deploy ceph.
I had the exact same issue. The last piece in the cleanup puzzle was to delete dangling docker volumes on all nodes:
docker volume rm $(docker volume ls -qf dangling=true)
Purge Ceph
The following will remove all remains of Ceph (after that I could install). This assumes that the nodename is reachable via ssh.
helm del --purge ceph
kubectl delete namespaces ceph
while read host; do
echo $host; ssh -n $host -l root -- rm -rf /var/lib/ceph-helm /etc/ceph
done < <(kubectl get nodes -o name|sed 's!^node/!!')
I also rebooted all nodes (not sure if this is a must).
Verify that kubernetes is gone:
kubectl get all --all-namespaces -o name|grep ceph
EDIT: Still fails :-(
Deleting the OSD pods (works!)
After deleting all OSD pods the cluster came up:
kubectl delete pod -n ceph ceph-osd-dev-vdb-m28wr
...
the same problem. when I run make: find: ‘secrets’: No such file or directory
I have same problem.
ceph-rgw-keyring-generator job create pod ，alway run failed 。
This causes other created pods to fail all the time
I've tried purge twice, but it's the same problem
I have same problem.
ceph-rgw-keyring-generatorjob create pod ，alway run failed 。 This causes other created pods to fail all the timeI've tried purge twice, but it's the same problem
the same, @xinghui-hou, did you find the solution?
