kube-monkey
kube-monkey copied to clipboard
Published
20 hours ago •
asobti
asobti
No terminations scheduled problem
Hello, I am facing similar issue which you already close it before #80. However, I am working on it from past few days and I am not able to solve it.
I0721 19:41:07.985992 1 config.go:82] Successfully validated configs
I0721 19:41:07.986015 1 main.go:54] Starting kube-monkey with v logging level 5 and local log directory /var/log/kube-monkey
I0721 19:41:08.011412 1 kubemonkey.go:19] Debug mode detected!
I0721 19:41:08.011452 1 kubemonkey.go:20] Status Update: Generating next schedule in 300 sec
I0721 19:46:08.011598 1 schedule.go:64] Status Update: Generating schedule for terminations
I0721 19:46:08.179514 1 schedule.go:57] Status Update: 0 terminations scheduled today
********** Today's schedule **********
No terminations scheduled
********** End of schedule **********
I0721 19:46:08.179589 1 kubemonkey.go:62] Status Update: Waiting to run scheduled terminations.
I0721 19:46:08.179603 1 kubemonkey.go:76] Status Update: All terminations done.
Below are my config file for more details:
-
configmap.yaml
--- apiVersion: v1 kind: ConfigMap metadata: name: kube-monkey-config-map namespace: kube-system data: config.toml: | [kubemonkey] run_hour = 8 start_hour = 10 end_hour = 16 grace period_sec=120 blacklisted_namespaces = ["kube-system"] whitelisted_namespaces=["default"] time_zone="Canada/Eastern" [debug] enabled = true schedule_delay=300 force_should_kill = true schedule_immediate_kill = true -
deployment.yaml
--- apiVersion: apps/v1 kind: Deployment metadata: name: kube-monkey namespace: kube-system spec: replicas: 1 selector: matchLabels: app: kube-monkey template: metadata: labels: app: kube-monkey spec: serviceAccount:default serviceAccountName:default containers: - name: kube-monkey command: - "/kube-monkey" args: ["-v=5", "-log_dir=/var/log/kube-monkey"] image: ayushsobti/kube-monkey:v0.3.0 env: - name: KUBE_MONKEY_ID value: CLUSTER_A volumeMounts: - name: config-volume mountPath: "/etc/kube-monkey" volumes: - name: config-volume configMap: name: kube-monkey-config-map -
docker-registry yaml(Victim application yaml)
apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: creationTimestamp: '2021-07-14T18:09:09Z' generation: 6 labels: docker-registry: default kube-monkey/enabled: enabled kube-monkey/identifier: docker-registry kube-monkey/kill-mode: fixed kube-monkey/kill-value: '1' kube-monkey/mtbf: '1' name: docker-registry namespace: default resourceVersion: '313891' selfLink: >- /apis/apps.openshift.io/v1/namespaces/default/deploymentconfigs/docker-registry uid: 9604c68c-e4ce-11eb-bd40-08002774d158 spec: replicas: 3 revisionHistoryLimit: 10 selector: docker-registry: default strategy: activeDeadlineSeconds: 21600 resources: {} rollingParams: intervalSeconds: 1 maxSurge: 25% maxUnavailable: 25% timeoutSeconds: 600 updatePeriodSeconds: 1 type: Rolling template: metadata: creationTimestamp: null labels: docker-registry: default kube-monkey/enabled: enabled kube-monkey/identifier: docker-registry kube-monkey/kill-mode: fixed kube-monkey/kill-value: '1' kube-monkey/mtbf: '1' spec: containers: - env: - name: REGISTRY_HTTP_ADDR value: ':5000' - name: REGISTRY_HTTP_NET value: tcp - name: REGISTRY_HTTP_SECRET value: t84he+iqXzoG4l3QNS4Y02m4TxV2P7Ww53k/BvoNZKU= - name: REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA value: 'false' image: 'openshift/origin-docker-registry:v3.11.0' imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 name: registry ports: - containerPort: 5000 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5000 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: requests: cpu: 100m memory: 256Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /registry name: registry-storage dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: registry serviceAccountName: registry terminationGracePeriodSeconds: 30 volumes: - hostPath: path: /var/lib/minishift/base/openshift.local.pv/registry type: '' name: registry-storage test: false triggers: - type: ConfigChange status: availableReplicas: 3 conditions: - lastTransitionTime: '2021-07-20T20:07:16Z' lastUpdateTime: '2021-07-20T20:07:16Z' message: Deployment config has minimum availability. status: 'True' type: Available - lastTransitionTime: '2021-07-20T21:01:15Z' lastUpdateTime: '2021-07-20T21:01:18Z' message: replication controller "docker-registry-4" successfully rolled out reason: NewReplicationControllerAvailable status: 'True' type: Progressing details: causes: - type: ConfigChange message: config change latestVersion: 4 observedGeneration: 6 readyReplicas: 3 replicas: 3 unavailableReplicas: 0 updatedReplicas: 3
I have also tried with same configmap.yaml that Kube-monkey repository has but hit the wall.
I have deploy Kube-monkey on OCP 4. It is using Kubernetes Version 1.19.0. Note: I have also tried same thing with OpenShift 3.11.
Could anyone please help me with this problem ?
Hi @shreya8887! I'm not familiar with openshift so I'm not sure if I'll be able to help much.
But you can see here that kube-monkey only targets apps/v1 Deployments but in your case you are using a apps.openshift.io/v1 DeploymentConfig.
Does that create a regular deployment behind the scenes? And if so, does it carry over the labels?
It looks like someone tried to solve this before and that it requires code changes: https://github.com/faust64/kube-monkey/commit/6b9956dc6157b05387c0b1e5c569aa95e6903310