kube-controller-manager跟kube-apiserver连接认证报错Authentication is disabled

[guleng]

非常感谢 @opsnull 分享的文档 我搭建master高可用的时候发现kube-controller-manager起来后有个Authentication is disabled #---------------------------------------------------------- [root@k8s-master2 ssl]# systemctl status kube-controller-manager.service ● kube-controller-manager.service - Kubernetes Controller Manager Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2018-08-11 00:45:18 CST; 1 day 1h ago Docs: https://github.com/GoogleCloudPlatform/kubernetes Main PID: 3448 (kube-controller) CGroup: /system.slice/kube-controller-manager.service └─3448 /opt/kubernetes/kube-controller-manager --port=0 --secure-port=10252 --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig --service-cluster-ip-range=172.24.0.0/16 --cluster-...

Aug 11 00:45:18 3448 flags.go:27] FLAG: --tls-sni-cert-key="[]" Aug 11 00:45:18 3448 flags.go:27] FLAG: --unhealthy-zone-threshold="0.55" Aug 11 00:45:18 3448 flags.go:27] FLAG: --use-service-account-credentials="true" Aug 11 00:45:18 3448 flags.go:27] FLAG: --v="2" Aug 11 00:45:18 3448 flags.go:27] FLAG: --version="false" Aug 11 00:45:18 3448 flags.go:27] FLAG: --vmodule="" Aug 11 00:45:18 3448 controllermanager.go:116] Version: v1.10.4 Aug 11 00:45:18 3448 authentication.go:55] Authentication is disabled Aug 11 00:45:18 3448 serve.go:96] Serving securely on 127.0.0.1:10252 Aug 11 00:45:18 3448 leaderelection.go:175] attempting to acquire leader lease kube-system/kube-controller-manager... #---------------------------------------------------------- [root@k8s-master1 ssl]# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"} #-------------------------------------------------------- [root@k8s-master1 ssl]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master1_9d0106e9-9d94-11e8-b8f3-00163e065eb5","leaseDurationSeconds":15,"acquireTime":"2018-08-11T18:30:40Z","renewTime":"2018-08-11T18:34:15Z","leaderTransitions":16}' creationTimestamp: 2018-08-09T16:31:27Z name: kube-controller-manager namespace: kube-system resourceVersion: "216520" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: a9b29869-9bf1-11e8-8a78-00163e0ccf78 #-------------------------------------------------------- [root@k8s-master3 kubernetes]# vim /etc/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service] ExecStart=/opt/kubernetes/kube-controller-manager
--port=0
--secure-port=10252
--bind-address=127.0.0.1
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
--service-cluster-ip-range=172.24.0.0/16
--cluster-name=kubernetes
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
--experimental-cluster-signing-duration=8760h
--root-ca-file=/etc/kubernetes/ssl/ca.pem
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem
--leader-elect=true
--feature-gates=RotateKubeletServerCertificate=true
--controllers=*,bootstrapsigner,tokencleaner
--horizontal-pod-autoscaler-use-rest-clients=true
--horizontal-pod-autoscaler-sync-period=10s
--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem
--tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem
--use-service-account-credentials=true
--alsologtostderr=true
--insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers
--logtostderr=false
--log-dir=/var/log/kubernetes
--v=2 Restart=on Restart=on-failure RestartSec=5

[Install] WantedBy=multi-user.target #-------------------------------------------------------------------- kubectl config set-cluster kubernetes
--certificate-authority=/etc/kubernetes/ssl/ca.pem
--embed-certs=true
--server=https://10.10.140.122:8443
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager
--client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem
--client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem
--embed-certs=true
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager
--cluster=kubernetes
--user=system:kube-controller-manager
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig #-------------------------------------------------------------------- [root@k8s-master3 kubernetes]# cat kube-controller-manager.k8s-master3.unknownuser.log.WARNING.20180812-023849.5071 Log file created at: 2018/08/12 02:38:49 Running on machine: k8s-master3 Binary: Built with gc go1.9.3 for linux/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg W0812 02:38:49.704868 5071 authentication.go:55] Authentication is disabled E0812 02:39:19.705554 5071 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://10.10.140.122:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 10.10.140.122:6443: i/o timeout E0812 02:39:53.488181 5071 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://10.10.140.122:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 10.10.140.122:6443: i/o timeout #---------------------------------------------------------------------- [root@k8s-master3 kubernetes]# vim /var/log/messages Aug 12 02:41:57 k8s-master3 kube-apiserver: I0812 02:41:57.504283 752 logs.go:49] http: TLS handshake error from 100.97.205.129:46368: read tcp 10.10.10.13:6443->100.97.205.129:46368: read: connection reset by peer Aug 12 02:41:57 k8s-master3 kube-apiserver: I0812 02:41:57.538190 752 logs.go:49] http: TLS handshake error from 100.109.252.2:47256: read tcp 10.10.10.13:6443->100.109.252.2:47256: read: connection reset by peer #---------------------------------------------------------------------- 以上是kube-controller-manager的报错信息及配置文件 #---------------------------------------------------------------------- vim /etc/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target

[Service] ExecStart=/opt/kubernetes/kube-apiserver
--enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
--anonymous-auth=false
--experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml
--advertise-address=10.10.10.13
--bind-address=10.10.10.13
--insecure-port=0
--authorization-mode=Node,RBAC
--runtime-config=api/all
--enable-bootstrap-token-auth
--service-cluster-ip-range=172.24.0.0/16
--service-node-port-range=1-65535
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem
--client-ca-file=/etc/kubernetes/ssl/ca.pem
--kubelet-client-certificate=/etc/kubernetes/ssl/kubernetes.pem
--kubelet-client-key=/etc/kubernetes/ssl/kubernetes-key.pem
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem
--etcd-cafile=/etc/kubernetes/ssl/ca.pem
--etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem
--etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem
--etcd-servers=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379
--enable-swagger-ui=true
--allow-privileged=true
--apiserver-count=3
--audit-log-maxage=30
--audit-log-maxbackup=3
--audit-log-maxsize=100
--audit-log-path=/var/log/kube-apiserver-audit.log
--event-ttl=1h
--alsologtostderr=true
--logtostderr=false
--log-dir=/var/log/kubernetes
--v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536

[Install] WantedBy=multi-user.target #-------------------------------------------------------------------- 请大家帮忙看一下这个问题出在哪里，我看了很多次重新搭建几次都是这样，无招了请大家帮忙看一下，谢谢！

[opsnull]

看起来都正常，选举产生的 master 是 k8s-master1。只要按照文档中的检查 controller-manager 运行状态、metrics 都正常就 OK。

[opsnull]

BTW，Authentication is disabled 不需要 care。

[guleng]

@opsnull 有这种报错

systemctl status kube-controller-manager.service error retrieving resource lock kube-system/kube-controller-manager: Get https://10.10.140.122:6443/api/v1/namespaces/kube-sys...rk is unreachable Failed to parse service restart specifier, ignoring: on Failed to parse service restart specifier, ignoring: on Failed to parse service restart specifier, ignoring: on http: TLS handshake error from 127.0.0.1:42124: tls: first record does not look like a TLS handshake http: TLS handshake error from 127.0.0.1:42132: tls: first record does not look like a TLS handshake http: TLS handshake error from 127.0.0.1:42146: tls: first record does not look like a TLS handshake http: TLS handshake error from 127.0.0.1:42160: tls: first record does not look like a TLS handshake #--------------------------------------------------------------- systemctl status kube-apiserver.service http: TLS handshake error from 100.97.204.129:63057: read tcp 10.10.10.11:6443->100.97.204.129:63057: read: connection reset by peer http: TLS handshake error from 100.109.252.3:13307: read tcp 10.10.10.11:6443->100.109.252.3:13307: read: connection reset by peer http: TLS handshake error from 100.97.205.131:24216: read tcp 10.10.10.11:6443->100.97.205.131:24216: read: connection reset by peer http: TLS handshake error from 100.97.204.2:36470: read tcp 10.10.10.11:6443->100.97.204.2:36470: read: connection reset by peer http: TLS handshake error from 100.109.252.129:34784: read tcp 10.10.10.11:6443->100.109.252.129:34784: read: connection reset by peer

[guleng]

[root@k8s-master1 yum.repos.d]# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}

[typ431127]

我现在跟楼上报错一样

[root@node6 tmp]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02" scheduler Healthy ok etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"} etcd-3 Healthy {"health":"true"}

[kermit-ye]

同样这个问题 NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02" scheduler Healthy ok

只有一个节点curl有返回，其他两个节点curl没反应

[dianwen119]

kubectl get cs 中 controller-manager 检查健康的方式是通过http，但是kube-controller-manager.service 里面配置了 --port=0 \关闭监听 http /metrics 的请求 --secure-port=10252 \secure-port 直译就是安全端口嘛，即是https 。

既然http 都没有打开，怎么检查的到呢。

[xulis]

APIServer进行心跳检测是通过非安全(HTTP)端口进行的,代码在这里: https://github.com/kubernetes/kubernetes/blob/2da49321e62dfe8485282f691659e46a6d0c5aab/pkg/registry/core/rest/storage_core.go#L253 所以,对于这个问题,最简单的解决方案是,开放controller-manager的HTTP端口,且因为其默认端口是10252,所以在这里我们需要修改我们的HTTPS ( --secure-port)端口,或者你把它这个配置注释掉, 把配置文件中: --port=0 --secure-port=10252 两项去掉,去掉之后,服务默认开启HTTP -> 10252 , HTTPS -> 10257 另外,遇到问题提Issue请不要@项目作者,避免作者工作量太大, 开源项目是所有人一起维护. 祝工作顺利.

[lisanmengmeng]

我也遇到类似问题,版本：v1.20 kube-controller-manager 报错 E0316 17:32:54.064404 43162 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized

kube-apiserver 报错 E0316 17:32:54.062550 32576 authentication.go:53] Unable to authenticate the request due to an error: verifying certificate SN=625631038904098576053887334173004061427148865761, SKID=AE:71:63:8C:07:BE:8B:82:9A:7E:79:D7:9A:D8:91:1E:51:27:15:48, AKID=3D:ED:9F:9A:EC:7D:1B:A9:5C:A3:4B:83:EF:4C:EE:7E:17:8A:7A:E7 failed: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

[jiangxiaobin96]

有解决的办法了吗？