ks-installer icon indicating copy to clipboard operation
ks-installer copied to clipboard

Published 20 hours ago • verified publisher icon kubesphere

connection refused to kube-apiserver and ks-installer needs a restart

Open stoneshi-yunify opened this issue 2 years ago • 1 comments

I have a k3s (kubesphere lightweight cluster) and I was trying to install kubesphere v3.3.0 on it.

the k3s is not very stable, it restarts sometimes, and the ks-installer seems hold a stale connection with apiserver. the ks-installer logs a lot of connection refused errors even the k3s apiserver can be connected to.

I deleted the ks-installer pod and new ks-installer goes well, which proves the k3s apiserver has been recovered and old ks-installer pod was holding a bad connection.

ks-installer logs

**************************************************
Collecting installation results ...
E0818 08:29:49.319121       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:29:51.335045       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused


Task 'monitoring' failed:
******************************************************************************************************************************************************
{
 "counter": 192,
 "created": "2022-08-18T08:29:47.191095",
 "end_line": 181,
 "event": "runner_on_failed",
 "event_data": {
   "duration": 51.807748,
   "end": "2022-08-18T08:29:47.190962",
   "event_loop": null,
   "host": "localhost",
   "ignore_errors": null,
   "play": "localhost",
   "play_pattern": "localhost",
   "play_uuid": "fe1c257d-efb2-05c1-7329-000000000005",
   "playbook": "/kubesphere/playbooks/monitoring.yaml",
   "playbook_uuid": "3b1308ea-b6b9-48df-8ce0-8fc458930465",
   "remote_addr": "127.0.0.1",
   "res": {
     "_ansible_no_log": false,
     "attempts": 5,
     "changed": true,
     "cmd": "/usr/local/bin/kubectl patch cc ks-installer --type merge -p '{\"status\": {\"monitoring\": {\"status\": \"enabled\", \"enabledTime\": \"2022-08-18T08:28:55UTC\"}}}' -n kubesphere-system\n",
     "delta": "0:00:01.477809",
     "end": "2022-08-18 08:29:47.078739",
     "failed_when_result": true,
     "invocation": {
       "module_args": {
         "_raw_params": "/usr/local/bin/kubectl patch cc ks-installer --type merge -p '{\"status\": {\"monitoring\": {\"status\": \"enabled\", \"enabledTime\": \"2022-08-18T08:28:55UTC\"}}}' -n kubesphere-system\n",
         "_uses_shell": true,
         "argv": null,
         "chdir": null,
         "creates": null,
         "executable": null,
         "removes": null,
         "stdin": null,
         "stdin_add_newline": true,
         "strip_empty_ends": true,
         "warn": true
       }
     },
     "msg": "non-zero return code",
     "rc": 1,
     "start": "2022-08-18 08:29:45.600930",
     "stderr": "The connection to the server 10.101.244.194:443 was refused - did you specify the right host or port?",
     "stderr_lines": [
       "The connection to the server 10.101.244.194:443 was refused - did you specify the right host or port?"
     ],
     "stdout": "",
     "stdout_lines": []
   },
   "resolved_action": "command",
   "role": "ks-monitor",
   "start": "2022-08-18T08:28:55.383214",
   "task": "Monitoring | Importing ks-monitoring status",
   "task_action": "command",
   "task_args": "",
   "task_path": "/kubesphere/installer/roles/ks-monitor/tasks/main.yaml:17",
   "task_uuid": "fe1c257d-efb2-05c1-7329-00000000006e",
   "uuid": "ed102a84-905f-45b0-958f-6dbbeef03040"
 },
 "parent_uuid": "fe1c257d-efb2-05c1-7329-00000000006e",
 "pid": 2832,
 "runner_ident": "monitoring",
 "start_line": 180,
 "stdout": "fatal: [localhost]: FAILED! => {\"attempts\": 5, \"changed\": true, \"cmd\": \"/usr/local/bin/kubectl patch cc ks-installer --type merge -p '{\\\"status\\\": {\\\"monitoring\\\": {\\\"status\\\": \\\"enabled\\\", \\\"enabledTime\\\": \\\"2022-08-18T08:28:55UTC\\\"}}}' -n kubesphere-system\\n\", \"delta\": \"0:00:01.477809\", \"end\": \"2022-08-18 08:29:47.078739\", \"failed_when_result\": true, \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2022-08-18 08:29:45.600930\", \"stderr\": \"The connection to the server 10.101.244.194:443 was refused - did you specify the right host or port?\", \"stderr_lines\": [\"The connection to the server 10.101.244.194:443 was refused - did you specify the right host or port?\"], \"stdout\": \"\", \"stdout_lines\": []}",
 "uuid": "ed102a84-905f-45b0-958f-6dbbeef03040"
}
******************************************************************************************************************************************************
Failed to ansible-playbook ks-config.yaml
E0818 08:30:07.655115       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:09.671090       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:11.687109       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:13.703136       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:15.723120       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:17.735100       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:19.751071       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:21.767077       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:23.783111       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:25.799067       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:27.815017       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:29.831068       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:31.847042       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:33.867041       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:35.883031       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:37.900504       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused
E0818 08:30:39.911006       1 reflector.go:131] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0": dial tcp 10.101.244.194:443: connect: connection refused

10.101.244.194:443 is k3s apiserver.

log in ks-installer pod and curl the apiserver, succeeded:

bash-5.1$ curl -k -v "https://10.101.244.194:443/apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0"
*   Trying 10.101.244.194:443...
* Connected to 10.101.244.194 (10.101.244.194) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Aug 18 08:22:28 2022 GMT
*  expire date: Aug 18 08:30:39 2023 GMT
*  issuer: CN=k3s-server-ca@1660810948
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0]
* h2h3 [:scheme: https]
* h2h3 [:authority: 10.101.244.194]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fe9a456e1a0)
> GET /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0 HTTP/2
> Host: 10.101.244.194
> user-agent: curl/7.83.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 1000)!
< HTTP/2 403
< audit-id: f09bcbf9-7aa2-4339-b129-533137d26e7b
< audit-id: 02a714e7-06bf-4849-ba27-8752d308f2bb
< cache-control: no-cache, private
< cache-control: no-cache, private
< content-type: application/json
< date: Fri, 19 Aug 2022 01:16:58 GMT
< x-content-type-options: nosniff
< x-kubernetes-pf-flowschema-uid: 04b4c81d-de64-4ac4-9a62-9c3cbd823e3d
< x-kubernetes-pf-prioritylevel-uid: 172d57cb-3723-4b70-9963-eaee46154de2
< content-length: 495
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "clusterconfigurations.installer.kubesphere.io \"ks-installer\" is forbidden: User \"system:anonymous\" cannot list resource \"clusterconfigurations\" in API group \"installer.kubesphere.io\" in the namespace \"kubesphere-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "ks-installer",
    "group": "installer.kubesphere.io",
    "kind": "clusterconfigurations"
  },
  "code": 403
* Connection #0 to host 10.101.244.194 left intact
}bash-5.1$

restarted ks-installer pod then everything goes well.

Is there a way to solve the problem without restarting the ks-installer ?

stoneshi-yunify avatar Aug 19 '22 01:08 stoneshi-yunify 

$ kubectl --kubeconfig=vv.config version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:21:56Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3+k3s1", GitCommit:"5fb370e53e0014dc96183b8ecb2c25a61e891e76", GitTreeState:"clean", BuildDate:"2022-01-27T02:12:21Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

ks: v3.3.0

stoneshi-yunify avatar Aug 19 '22 01:08 stoneshi-yunify