follow-me-install-kubernetes-cluster
follow-me-install-kubernetes-cluster copied to clipboard
opsnull
x509: subject with cn=system:kube-controller-manager is not in the allowed list: []
按照： 09-4.metrics-server插件这篇说明 ，在增加kube-apiserver.service 增加以下：
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem
--requestheader-allowed-names=""
--requestheader-extra-headers-prefix="X-Remote-Extra-"
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem
--proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem
--runtime-config=api/all=true
kube-controllr-manager.service 增加：
--horizontal-pod-autoscaler-use-rest-clients=true
重启apiserver和controller-manager之后，在kube-apiserver的日志一直有错误（x509)：
Jun 29 03:04:23test-01 kube-apiserver[28047]: I0629 03:04:23.241329 28047 wrap.go:42] GET /apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations: (4.365848ms) 200 [[kube-apiserver/v1.10.5 (linux/amd64) kubernetes/32ac1c9] 192.168.0.4:53784]
Jun 29 03:04:23test-01 kube-apiserver[28047]: I0629 03:04:23.344236 28047 wrap.go:42] GET /apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations: (4.9489ms) 200 [[kube-apiserver/v1.10.5 (linux/amd64) kubernetes/32ac1c9] 192.168.0.4:53784]
Jun 29 03:04:23test-01 kube-apiserver[28047]: W0629 03:04:23.496582 28047 x509.go:172] x509: subject with cn=system:kube-scheduler is not in the allowed list: []
Jun 29 03:04:23test-01 kube-apiserver[28047]: I0629 03:04:23.500949 28047 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (4.548608ms) 200 [[kube-scheduler/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 192.168.0.4:53578]
Jun 29 03:04:23test-01 kube-apiserver[28047]: W0629 03:04:23.503045 28047 x509.go:172] x509: subject with cn=system:kube-scheduler is not in the allowed list: []
Jun 29 03:04:23test-01 kube-apiserver[28047]: I0629 03:04:23.511143 28047 wrap.go:42] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (8.422426ms) 200 [[kube-scheduler/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 192.168.0.4:53578]
Jun 29 03:04:24test-01 kube-apiserver[28047]: W0629 03:04:24.040579 28047 x509.go:172] x509: subject with cn=system:kube-controller-manager is not in the allowed list: []
Jun 29 03:04:24test-01 kube-apiserver[28047]: I0629 03:04:24.044569 28047 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (4.896406ms) 200 [[kube-controller-manager/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 192.168.0.4:53212]
Jun 29 03:04:24test-01 kube-apiserver[28047]: W0629 03:04:24.046287 28047 x509.go:172] x509: subject with cn=system:kube-controller-manager is not in the allowed list: []
Jun 29 03:04:24test-01 kube-apiserver[28047]: I0629 03:04:24.053249 28047 wrap.go:42] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (6.823982ms) 200 [[kube-controller-manager/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 192.168.0.4:53212]
证书也是按照：创建 metrics-server 证书签名请求 做的证书。
{
"CN": "aggregator",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
metrcs-server 输出的 metrics ：
- 直接使用 kubectl 命令访问，会报错：
$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1" | jq .
Error from server (ServiceUnavailable): the server is currently unable to handle the request
- 使用：kubectl proxy方式访问，是没问题的。
不知道是问题出在哪里？为什么直接使用 kubectl 命令访问，会报错？kube-apiserver里面的x509警告是哪里导致的？
从源码里复制出来的resource-reader.yaml这个文件中设置metric-server可访问apiserver的资源的授权有写问题造成的.
解决方法, 修改resource-reader.yaml:
注意下面的注释说明
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- metrics.k8s.io ########这里将原来的""换成 "metrics.k8s.io"
resources:
- pods
- nodes
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- deployments
verbs:
- get
- list
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
#- kind: ServiceAccount
- kind: User ############## 把ServiceAccount改为User
name: metrics-server ######### 创建metric-server证书的时候, 将CN写成这里的"metrics-server"
namespace: kube-system
注意 apiserver的启动参数--requestheader-allowed-names 留空"" 代表允许任何用户访问apiserver(授权是另外一回事, 这个和RBAC没啥关系) 在这里可以留空或者写成ClusterRoleBinding里subject里的用户名
最后 kubectl apply -f resource-reader.yaml
测试方法 kubectl top pods
[k8s@kube-node1 metrics-server]$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
my-nginx-86555897f9-57v4t 0m 2Mi
my-nginx-86555897f9-8mzxq 0m 2Mi
my-nginx-86555897f9-dkm8m 0m 2Mi
nginx-ds-l8qkg 0m 2Mi
nginx-ds-sll9g 0m 2Mi
php-59447fb5c-28dwl 0m 10Mi
php-59447fb5c-g6pwr 0m 9Mi
php-59447fb5c-phtrh 0m 9Mi
--requestheader-allowed-names=aggregator 只要不设置该参数即可，也不会再出现相关报错。不是设置空，是删掉该参数
官方文档中对于该参数的解释 --requestheader-allowed-names stringSlice | List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
@miaoxiaoy 我按照你的方法，还是有问题：
$ kubectl logs metrics-server-v0.2.1-66b95ddf44-2th8c -n kube-system -c metrics-server
E0725 06:20:30.577090 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:30.615137 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0725 06:20:31.581478 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:31.617166 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0725 06:20:32.597102 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:32.619230 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
@4220182 heapster源码里的授权文件权限有点问题
cat /home/k8s/k8s-install/heapster-1.5.3/deploy/kube-config/rbac/heapster-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster-kubelet-api
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kubelet-api-admin
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system
原因是system:kubelet-api-admin这个clusterrole角色的权限不足, 命令行直接修改system:kubelet-api-admin这个集群角色
[k8s@kube-node1 hpa]$ kubectl edit clusterrole system:kubelet-api-admin
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-04T11:17:42Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kubelet-api-admin
resourceVersion: "1989133"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Akubelet-api-admin
uid: debcb355-7f7b-11e8-b7a6-005056860efb
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- proxy
- apiGroups:
- ""
resources:
- nodes/log
- nodes/metrics
- nodes/proxy
- nodes/spec
- nodes/stats
verbs:
- '*'
#################################### 下面是新增的, 增加了对pod, namespace的 get, list, watch 权限
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
从源码里复制出来的resource-reader.yaml这个文件中设置metric-server可访问apiserver的资源的授权有写问题造成的.
解决方法, 修改resource-reader.yaml:
注意下面的注释说明
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:metrics-server labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile rules: - apiGroups: - metrics.k8s.io ########这里将原来的""换成 "metrics.k8s.io" resources: - pods - nodes - namespaces verbs: - get - list - watch - apiGroups: - "extensions" resources: - deployments verbs: - get - list - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:metrics-server labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: #- kind: ServiceAccount - kind: User ############## 把ServiceAccount改为User name: metrics-server ######### 创建metric-server证书的时候, 将CN写成这里的"metrics-server" namespace: kube-system注意 apiserver的启动参数--requestheader-allowed-names 留空"" 代表允许任何用户访问apiserver(授权是另外一回事, 这个和RBAC没啥关系) 在这里可以留空或者写成ClusterRoleBinding里subject里的用户名
最后 kubectl apply -f resource-reader.yaml
测试方法 kubectl top pods
[k8s@kube-node1 metrics-server]$ kubectl top pods NAME CPU(cores) MEMORY(bytes) my-nginx-86555897f9-57v4t 0m 2Mi my-nginx-86555897f9-8mzxq 0m 2Mi my-nginx-86555897f9-dkm8m 0m 2Mi nginx-ds-l8qkg 0m 2Mi nginx-ds-sll9g 0m 2Mi php-59447fb5c-28dwl 0m 10Mi php-59447fb5c-g6pwr 0m 9Mi php-59447fb5c-phtrh 0m 9Mi
My metrics-server setup without heapster, and it can work fine. But, after following your steps, metrics-server cannot work, and the warnings in kube-apiserver still exist.
kubectl edit clusterrole system:kubelet-api-admin
按照您说的 执行 $ kubectl edit clusterrole system:kubelet-api-admin 修改权限后，确实可以了. /apis/metrics.k8s.io/v1beta1/nodes /apis/metrics.k8s.io/v1beta1/pods 都有数据了。
但这种手动修改的权限， 重启后会不会重置呀? 有没有永久修改的方法呀
