How to replicate default kubernetes deployment services

[omidraha]

How to replicate default kubernetes deployment services with install options.

I installed k3s with 3 nodes using --cluster and --server options.

How is it possible to replicate these default services including coredns and metrics-server to run on each master nodes with install options?

Info:

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/

export USER=ubuntu
export SERVER_01=1.1.1.1
export SERVER_02=2.2.2.2
export SERVER_03=3.3.3.3

k3sup install --ip $SERVER_01 --user $USER --cluster --k3s-extra-args '--disable traefik --disable servicelb --disable local-storage --node-external-ip 1.1.1.1'
k3sup join --ip $SERVER_02  --user $USER --server-user $USER  --server-ip $SERVER_01 --server --k3s-extra-args '--disable traefik --disable servicelb --disable local-storage --node-external-ip 2.2.2.2'
k3sup join --ip $SERVER_03  --user $USER --server-user $USER  --server-ip $SERVER_01 --server --k3s-extra-args '--disable traefik --disable servicelb  --disable local-storage --node-external-ip 3.3.3.3'

info:

kubectl get nodes -A  -o wide

NAME             STATUS   ROLES                       AGE     VERSION        INTERNAL-IP   EXTERNAL-IP      OS-IMAGE       KERNEL-VERSION   CONTAINER-RUNTIME
ip-10-10-1-110   Ready    control-plane,etcd,master   4m39s   v1.28.4+k3s2   10.10.1.110   1.1.1.1    Ubuntu 23.10   6.5.0-1011-aws   containerd://1.7.7-k3s1
ip-10-10-1-120   Ready    control-plane,etcd,master   2m14s   v1.28.4+k3s2   10.10.1.120   2.2.2.2    Ubuntu 23.10   6.5.0-1011-aws   containerd://1.7.7-k3s1
ip-10-10-1-130   Ready    control-plane,etcd,master   46s     v1.28.4+k3s2   10.10.1.130   3.3.3.3    Ubuntu 23.10   6.5.0-1011-aws   containerd://1.7.7-k3s1

Info:

kubectl get pods -A  -o wide
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP          NODE             NOMINATED NODE   READINESS GATES
kube-system   coredns-6799fbcd5-nvv8j           1/1     Running   0          9m22s   10.42.0.3   ip-10-10-1-110   <none>           <none>
kube-system   metrics-server-67c658944b-brf4x   1/1     Running   0          9m22s   10.42.0.2   ip-10-10-1-110   <none>           <none>

Info:

kubectl get deploy -A  -o wide

NAMESPACE     NAME             READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS       IMAGES                                    SELECTOR
kube-system   coredns          1/1     1            1           8m51s   coredns          rancher/mirrored-coredns-coredns:1.10.1   k8s-app=kube-dns
kube-system   metrics-server   1/1     1            1           8m50s   metrics-server   rancher/mirrored-metrics-server:v0.6.3    k8s-app=metrics-server

[omidraha]

Also dns resolving only works on the cluster node, and not on the other nodes.

I tried fresh install with HA embedded db and also server, agent options.

The dns server address 10.43.0.10 is only accessible from cluster node ( by ssh to node) and pods on this cluster node (by running pods on this node). And is not accessible from pods on other nodes (by running pods on other nodes) or other nodes (by ssh to other nodes).

# Node 1
ubuntu@ip-10-10-1-110:~$ nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:09 UTC
Nmap scan report for 10.43.0.10
Host is up (0.00015s latency).

PORT   STATE SERVICE
53/tcp open  domain

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

# Node 2
ubuntu@ip-10-10-1-120:~$ nmap 10.43.0.10 -Pn -p53

Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:06 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.04 seconds

# Node 3
ubuntu@ip-10-10-1-130:~$  nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:10 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.05 seconds

[Comradin]

Hi,

coredns is installed as part of the control plane of k3s. As it is a deployment, you have to scale it after the installation.

kubectl get deployment -n kube-system -o go-template='{{range .items}}{{.metadata.name}}:{{"\n"}}{{.spec.replicas}}{{"\n"}}{{end}}'
coredns:
1
local-path-provisioner:
1
metrics-server:
1
traefik:
1

Haven't looked into the k3sup code itself, but I doubt it deploys software to the cluster. But if you use an install.sh file, like suggested in the README.md file, you could just add a line like this to it at the end, to control the amount of pods on your own:

kubectl scale deployment coredns -n kube-system --relicas 3