monasca
Kubernetes error with "kubernetes_get_host.py" in monasca-agent
Running the script as is and with monasca pike creates the following error:
raise exceptions.KubernetesAPIConnectionError(exception_message)
monasca_agent.common.exceptions.KubernetesAPIConnectionError: Could not get agent pod from Kubernetes API to get host IP with error - HTTPSConnectionPool(host='10.0.0.1', port=443): Max retries exceeded with url: //api/v1/namespaces/default/pods/monasca-agent-deployment-1746399571-4l84t (Caused by SSLError(CertificateError("hostname '10.0.0.1' doesn't match either of 'kubernetes.default.svc.cluster.local', 'kubernetes.default.svc', 'kubernetes.default', 'kubernetes', 'localhost', '192.168.99.101', '10.0.0.1'",),))
The kubernetes service in 1.7.5 is defined as:
$ kubectl describe service kubernetes
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.0.0.1
Port: https 443/TCP
TargetPort: 8443/TCP
Endpoints: 10.0.2.15:8443
Session Affinity: ClientIP
Events: <none>
This could be tied to https://github.com/kubernetes-incubator/client-python/issues/36 ?
The python script "kubernetes_get_host.py" used in monasca-agent can be replaced with the DownwardAPI (see https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) and the variable injected via en environment variable:
env:
- name: AGENT_HOSTNAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
I would propose to remove the python script and associated code in Dockerfile, and start.sh, etc and instead document how to inject the nodename.
If that is acceptable, I can prepare a fix.
I think kubernetes_get_host.py predates nodeName being available in the downward API, but it would be good to switch. Maybe @mhoppal can confirm?
That said, you're right, the cause of the error is probably due to outdated ipaddress and urllib3 packages (see also). Removing kubernetes_get_host.py will fix this issue and let the agent start, but the plugins that use the kubernetes API should still fail.
If you're getting this from the monasca/agent containers that's pretty surprising... no idea how we haven't hit that in the agent plugins.
Output from the latest agent-collector image shows compatible versions:
[tim:~]↥ 3s % docker run --rm=true -it monasca/agent-collector:master-20171121-122311 pip list | grep -E '(urllib3|ipaddress)'
ipaddress (1.0.18)
urllib3 (1.22)
The plugin is fixed IFF the ipaddress package is added to dependencies. At the moment ipaddress is NOT included.
I raised an issue on monasca story board for this. https://review.openstack.org/#/c/519651/
Sent from my iPhone
On 21 Nov 2017, at 18:58, Tim Buckley [email protected] wrote:
Output from the latest agent-collector image shows compatible versions:
[tim:~]↥ 3s % docker run --rm=true -it monasca/agent-collector:master-20171121-122311 pip list | grep -E '(urllib3|ipaddress)' ipaddress (1.0.18) urllib3 (1.22) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Kubernetes_get_host.py is still included in the monasca agent base image.
So the problem remains and using the downward API would be a safer option.
On Wed, 17 Jan 2018 at 10:48, Witek Bedyk [email protected] wrote:
The upstream bugfix has been merged. Are any actions still required here?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/monasca/monasca-docker/issues/327#issuecomment-358251817, or mute the thread https://github.com/notifications/unsubscribe-auth/AdweXApYWaG0zkA8jeLB1yVekHo43CYGks5tLcHWgaJpZM4Qc-9D .