kubernetes-vmware-autoscaler
kubernetes-vmware-autoscaler copied to clipboard
Fred78290
What value should be set in sample?
Hello.
I want to deploy sample, but I don't understand some configmap value between <>. That's as follows.
address: <YOUR_GRPC_SERVER_FQCN_OR_IP>
port: <YOUR_GRPC_SERVER_PORT>
identifier: <SECRET/IDENTIFIER SHARED BETWEEN CLIENT_SERVER>
I guess to address is vcenter IP address,and what port, identifier?
And the following is everything to configmap. Please help to check if the other value is correct.
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-config
data:
grpc.conf: |-
address: <YOUR_GRPC_SERVER_FQCN_OR_IP>
port: <YOUR_GRPC_SERVER_PORT>
identifier: <SECRET/IDENTIFIER SHARED BETWEEN CLIENT_SERVER>
timeout: 60
config:
kubeAdmAddress: 10.60.200.170
kubeAdmToken: 9w6o68.jbdndje84e6xc40s
kubeAdmCACert: "~~~~"
kubeAdmExtraArguments:
- --ignore-preflight-errors=All
Hello
Values depend how you deploy the vmware auto scaler.
If you use or read the outsided project autoscaled-masterkube-vmware, it will answer to your question, specially create-master-kube.sh
So if you use my autoscaler template. The file will be suppiled with these values:
{
"address": "unix:/var/run/cluster-autoscaler/vmware.sock",
"secret": "vmware",
"timeout": 300,
"config": {
"kubeAdmAddress": "X.Y.Z.A:6443",
"kubeAdmToken": ".....",
"kubeAdmCACert": "sha256:.....",
"kubeAdmExtraArguments": [
"--ignore-preflight-errors=All"
]
}
}
because it run inside a pod with 2 containers.
Regards
I am also confused about the following para: address: <YOUR_GRPC_SERVER_FQCN_OR_IP> port: <YOUR_GRPC_SERVER_PORT> identifier: <SECRET/IDENTIFIER SHARED BETWEEN CLIENT_SERVER> how about this related to vsphere?
I think this repo only enhance cluster autoscaler to use grpc cloud provider, but this repo don't provide this grpc cloud provider?
I understand that vsphere cloud provider can use helm to deploy: https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-helm.md
Hello
Values depend how you deploy the vmware auto scaler.
If you use or read the outsided project autoscaled-masterkube-vmware, it will answer to your question, specially create-master-kube.sh
So if you use my autoscaler template. The file will be suppiled with these values:
{ "address": "unix:/var/run/cluster-autoscaler/vmware.sock", "secret": "vmware", "timeout": 300, "config": { "kubeAdmAddress": "X.Y.Z.A:6443", "kubeAdmToken": ".....", "kubeAdmCACert": "sha256:.....", "kubeAdmExtraArguments": [ "--ignore-preflight-errors=All" ] } }because it run inside a pod with 2 containers.
Regards how can I deploy the grpc cloud provider? I Change like above, but:
I think what I need is how to deploy the vsphere grpc cloud provider I try to deploy this: https://github.com/kubernetes/cloud-provider-vsphere/tree/release-1.22 but I seem can't connect to it..
Hi, Try first to use my other project autoscaled-masterkube-vmware as is.
Theses scripts will do the job for you
- Create a seed image with all kubernetes presetup
- Create config files need to run a vanilla kubernetes with kubeadm
- Create config files to configure autoscaler
The vmware autoscaler is a companion of my cloned autoscaler adding out of tree cluster provider support.
Hi, Try first to use my other project autoscaled-masterkube-vmware as is.
Theses scripts will do the job for you
- Create a seed image with all kubernetes presetup
- Create config files need to run a vanilla kubernetes with kubeadm
- Create config files to configure autoscaler
The vmware autoscaler is a companion of my cloned autoscaler adding out of tree cluster provider support.
thank you, I have read this project.. I seem a little complicated for me. what I want is I have a kubernetes cluster created by kubeadm, and I install your project--autoscaler, but I don't know how to run a grpc cloud provider? Is there any simple way?
As sample
Generated from create-masterkube.sh
deploy autoscaler
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: vsphere-autoscaler
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
-
apiGroups:
- ''
resources:
- events
- endpoints
verbs:
- create
- patch
-
apiGroups:
- ''
resources:
- pods/eviction
verbs:
- create
-
apiGroups:
- ''
resources:
- nodes/status
verbs:
- update
-
apiGroups:
- ''
resources:
- pods/status
verbs:
- update
-
apiGroups:
- ''
resources:
- endpoints
resourceNames:
- cluster-autoscaler
- vsphere-autoscaler
verbs:
- get
- update
- delete
-
apiGroups:
- ''
resources:
- nodes
verbs:
- watch
- list
- get
- update
- delete
- patch
-
apiGroups:
- ''
resources:
- pods
- namespaces
- services
- replicationcontrollers
- persistentvolumeclaims
- persistentvolumes
verbs:
- watch
- list
- get
- update
-
apiGroups:
- extensions
resources:
- replicasets
- daemonsets
verbs:
- watch
- list
- get
-
apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- watch
- list
-
apiGroups:
- apps
resources:
- statefulsets
- replicasets
- daemonsets
verbs:
- watch
- list
- get
- delete
-
apiGroups:
- storage.k8s.io
resources:
- storageclasses
- csinodes
verbs:
- get
- list
- watch
-
apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- watch
- list
- get
-
apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
-
apiGroups:
- coordination.k8s.io
resourceNames:
- cluster-autoscaler
resources:
- leases
verbs:
- get
- update
-
apiGroups:
- storage.k8s.io
resources:
- csistoragecapacities
- csidrivers
verbs:
- list
- get
- update
- watch
-
apiGroups:
- nodemanager.aldunelabs.com
resources:
- '*'
verbs:
- list
- get
- update
- watch
- delete
- patch
-
apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- list
- get
- update
- watch
- delete
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
-
apiGroups:
- ''
resources:
- configmaps
verbs:
- create
-
apiGroups:
- ''
resources:
- configmaps
resourceNames:
- cluster-autoscaler-status
verbs:
- delete
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-autoscaler
subjects:
-
kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
-
kind: ServiceAccount
name: vsphere-autoscaler
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-autoscaler
subjects:
-
kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
-
kind: ServiceAccount
name: vsphere-autoscaler
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
k8s-app: cluster-autoscaler
template:
metadata:
labels:
k8s-app: cluster-autoscaler
spec:
priorityClassName: system-cluster-critical
serviceAccountName: cluster-autoscaler
nodeSelector:
master: 'true'
tolerations:
-
key: node-role.kubernetes.io/master
effect: NoSchedule
-
key: node-role.kubernetes.io/control-plane
effect: NoSchedule
initContainers:
-
image: busybox
name: cluster-autoscaler-init
command:
- /bin/sh
- -c
- rm -f /var/run/cluster-autoscaler/vmware.sock
volumeMounts:
-
name: cluster-socket
mountPath: /var/run/cluster-autoscaler
containers:
-
image: fred78290/vsphere-autoscaler:v1.25.2
name: vsphere-autoscaler
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
command:
- /usr/local/bin/vsphere-autoscaler
- --src-etcd-ssl-dir=/etc/etcd/ssl
- --dst-etcd-ssl-dir=/etc/etcd/ssl
- --config=/etc/cluster/kubernetes-vmware-autoscaler.json
- --save=/var/run/cluster-autoscaler/vmware-autoscaler-state.json
- --log-level=info
imagePullPolicy: Always
volumeMounts:
-
name: cluster-socket
mountPath: /var/run/cluster-autoscaler
-
name: config-cluster-autoscaler
mountPath: /etc/cluster
-
name: autoscaler-ssh-keys
mountPath: /root/.ssh
-
name: etcd-ssl
mountPath: /etc/etcd/ssl
-
name: kubernetes-pki
mountPath: /etc/kubernetes/pki
-
image: fred78290/cluster-autoscaler:v1.25.2
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
command:
- ./cluster-autoscaler
- --v=1
- --stderrthreshold=info
- --cloud-provider=grpc
- --cloud-config=/etc/cluster/grpc-config.json
- --nodes=0:9:true/vmware-ca-k8s
- --max-nodes-total=9
- --cores-total=0:16
- --memory-total=0:48
- --node-autoprovisioning-enabled
- --max-autoprovisioned-node-group-count=1
- --scale-down-enabled=true
- --scale-down-delay-after-add=1m
- --scale-down-delay-after-delete=1m
- --scale-down-delay-after-failure=1m
- --scale-down-unneeded-time=1m
- --scale-down-unready-time=1m
- --unremovable-node-recheck-timeout=1m
imagePullPolicy: Always
volumeMounts:
-
name: cluster-socket
mountPath: /var/run/cluster-autoscaler
-
name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
-
name: config-cluster-autoscaler
mountPath: /etc/cluster
readOnly: true
volumes:
-
name: cluster-socket
emptyDir: {}
-
name: config-cluster-autoscaler
configMap:
name: config-cluster-autoscaler
-
name: ssl-certs
hostPath:
path: /etc/ssl/certs/ca-certificates.crt
-
name: autoscaler-ssh-keys
secret:
secretName: autoscaler-ssh-keys
defaultMode: 384
-
name: etcd-ssl
secret:
secretName: etcd-ssl
defaultMode: 384
-
name: kubernetes-pki
configMap:
name: kubernetes-pki
config/vmware-ca-k8s/config/kubernetes-vmware-autoscaler.json
{
"use-external-etcd": false,
"src-etcd-ssl-dir": "/etc/etcd/ssl",
"dst-etcd-ssl-dir": "/etc/kubernetes/pki/etcd",
"kubernetes-pki-srcdir": "/etc/kubernetes/pki",
"kubernetes-pki-dstdir": "/etc/kubernetes/pki",
"network": "unix",
"listen": "/var/run/cluster-autoscaler/vmware.sock",
"secret": "vmware",
"minNode": 0,
"maxNode": 9,
"maxNode-per-cycle": 2,
"node-name-prefix": "autoscaled",
"managed-name-prefix": "managed",
"controlplane-name-prefix": "master",
"nodePrice": 0,
"podPrice": 0,
"image": "jammy-kubernetes-cni-flannel-v1.25.4-containerd-amd64",
"optionals": {
"pricing": false,
"getAvailableMachineTypes": false,
"newNodeGroup": false,
"templateNodeInfo": false,
"createNodeGroup": false,
"deleteNodeGroup": false
},
"kubeadm": {
"address": "192.168.1.20:6443",
"token": "AAA.BBBBB",
"ca": "sha256:......",
"extras-args": [
"--ignore-preflight-errors=All"
]
},
"default-machine": "large",
"machines": {
"tiny": {
"memsize": 2048,
"vcpus": 2,
"disksize": 10240
},
"small": {
"memsize": 4096,
"vcpus": 2,
"disksize": 20480
},
"medium": {
"memsize": 4096,
"vcpus": 4,
"disksize": 20480
},
"large": {
"memsize": 8192,
"vcpus": 4,
"disksize": 51200
},
"xlarge": {
"memsize": 16384,
"vcpus": 4,
"disksize": 102400
},
"2xlarge": {
"memsize": 16384,
"vcpus": 8,
"disksize": 102400
},
"4xlarge": {
"memsize": 32768,
"vcpus": 8,
"disksize": 102400
}
},
"node-labels": [
"topology.kubernetes.io/region=home",
"topology.kubernetes.io/zone=office",
"topology.csi.vmware.com/k8s-region=home",
"topology.csi.vmware.com/k8s-zone=office"
],
"cloud-init": {
"package_update": false,
"package_upgrade": false,
"runcmd": [
"echo 1 > /sys/block/sda/device/rescan",
"growpart /dev/sda 1",
"resize2fs /dev/sda1",
"echo '192.168.1.20 vmware-ca-k8s-masterkube vmware-ca-k8s-masterkube.example.com' >> /etc/hosts"
]
},
"ssh-infos": {
"user": "kubernetes",
"ssh-private-key": "/root/.ssh/id_rsa"
},
"vmware": {
"vmware-ca-k8s": {
"url": "https://[email protected]:[email protected]/sdk",
"uid": "[email protected]",
"password": "MyPassword",
"insecure": true,
"dc": "DC01",
"datastore": "datastore1",
"resource-pool": "APPLE/Resources/FR",
"vmFolder": "HOME",
"timeout": 300,
"template-name": "jammy-kubernetes-cni-flannel-v1.25.4-containerd-amd64",
"template": false,
"linked": false,
"customization": "",
"network": {
"domain": "example.com",
"dns": {
"search": [
"example.com"
],
"nameserver": [
"1.2.3.4"
]
},
"interfaces": [
{
"primary": false,
"exists": true,
"network": "VM Network",
"adapter": "vmxnet3",
"mac-address": "generate",
"nic": "eth0",
"dhcp": true,
"use-dhcp-routes": true,
"routes": [
{
"to": "W.X.Y.Z/16",
"via": "A.B.C.D",
"metric": 100
},
{
"to": "O.P.Q.R/8",
"via": "A.B.C.D",
"metric": 500
}
]
},
{
"primary": true,
"exists": true,
"network": "VM Private",
"adapter": "vmxnet3",
"mac-address": "generate",
"nic": "eth1",
"dhcp": true,
"use-dhcp-routes": false,
"address": "192.168.1.26",
"gateway": "10.0.0.1",
"netmask": "255.255.255.0",
"routes": []
}
]
}
}
}
}
grpc-config.json
{
"address": "unix:/var/run/cluster-autoscaler/vmware.sock",
"secret": "vmware",
"timeout": 300,
"config": {
"kubeAdmAddress": "192.168.1.20:6443",
"kubeAdmToken": "AAA.BBBB",
"kubeAdmCACert": "sha256:......",
"kubeAdmExtraArguments": [
"--ignore-preflight-errors=All"
]
}
}