kubernetes.core
kubernetes.core copied to clipboard
ansible-collections
Cannot be used with EKS
SUMMARY
I can't use the module to administer AWS EKS clusters
ISSUE TYPE
- Bug Report
COMPONENT NAME
kubernetes.core.k8s_exec
ANSIBLE VERSION
ansible 2.9.6
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/<my-user>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
COLLECTION VERSION
ansible-galaxy collection list don'work on my computer I had collections :
- kubernetes_core
- community_kubernetes
ll /home/<my-user>/.ansible/collections/ansible_collections/kubernetes/core/
total 248
drwxrwxr-x 8 <my-user> <my-user> 4096 mai 12 16:09 ./
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:09 ../
-rw------- 1 <my-user> <my-user> 72 mai 12 16:09 bindep.txt
-rw------- 1 <my-user> <my-user> 29092 mai 12 16:09 CHANGELOG.rst
drwxrwxr-x 2 <my-user> <my-user> 4096 mai 12 16:09 changelogs/
-rw------- 1 <my-user> <my-user> 107 mai 12 16:09 codecov.yml
-rw------- 1 <my-user> <my-user> 3819 mai 12 16:09 CONTRIBUTING.md
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:09 docs/
-rw------- 1 <my-user> <my-user> 91613 mai 12 16:09 FILES.json
drwxrwxr-x 2 <my-user> <my-user> 4096 mai 12 16:09 .github/
-rw------- 1 <my-user> <my-user> 329 mai 12 16:09 .gitignore
-rw------- 1 <my-user> <my-user> 35148 mai 12 16:09 LICENSE
-rw------- 1 <my-user> <my-user> 892 mai 12 16:09 Makefile
-rw------- 1 <my-user> <my-user> 1174 mai 12 16:09 MANIFEST.json
drwxrwxr-x 2 <my-user> <my-user> 4096 mai 12 16:09 meta/
drwxrwxr-x 10 <my-user> <my-user> 4096 mai 12 16:09 plugins/
-rw------- 1 <my-user> <my-user> 11816 mai 12 16:09 README.md
-rw------- 1 <my-user> <my-user> 47 mai 12 16:09 requirements.txt
-rw------- 1 <my-user> <my-user> 67 mai 12 16:09 setup.cfg
-rw------- 1 <my-user> <my-user> 56 mai 12 16:09 test-requirements.txt
drwxrwxr-x 5 <my-user> <my-user> 4096 mai 12 16:09 tests/
-rw------- 1 <my-user> <my-user> 889 mai 12 16:09 tox.ini
-rw------- 1 <my-user> <my-user> 305 mai 12 16:09 .yamllint
ll /home/<my-user>/.ansible/collections/ansible_collections/community/kubernetes/
total 136
drwxrwxr-x 7 <my-user> <my-user> 4096 mai 12 16:13 ./
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:13 ../
-rw------- 1 <my-user> <my-user> 36 mai 12 16:13 bindep.txt
-rw------- 1 <my-user> <my-user> 16344 mai 12 16:13 CHANGELOG.rst
drwxrwxr-x 2 <my-user> <my-user> 4096 mai 12 16:13 changelogs/
-rw------- 1 <my-user> <my-user> 107 mai 12 16:13 codecov.yml
-rw------- 1 <my-user> <my-user> 3311 mai 12 16:13 CONTRIBUTING.md
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:13 docs/
-rw------- 1 <my-user> <my-user> 5649 mai 12 16:13 FILES.json
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:13 .github/
-rw------- 1 <my-user> <my-user> 190 mai 12 16:13 .gitignore
-rw------- 1 <my-user> <my-user> 35148 mai 12 16:13 LICENSE
-rw------- 1 <my-user> <my-user> 1205 mai 12 16:13 Makefile
-rw------- 1 <my-user> <my-user> 1224 mai 12 16:13 MANIFEST.json
drwxrwxr-x 2 <my-user> <my-user> 4096 mai 12 16:13 meta/
-rw------- 1 <my-user> <my-user> 6396 mai 12 16:13 README.md
-rw------- 1 <my-user> <my-user> 35 mai 12 16:13 requirements.txt
-rw------- 1 <my-user> <my-user> 50 mai 12 16:13 setup.cfg
-rw------- 1 <my-user> <my-user> 20 mai 12 16:13 test-requirements.txt
drwxrwxr-x 3 <my-user> <my-user> 4096 mai 12 16:13 tests/
-rw------- 1 <my-user> <my-user> 264 mai 12 16:13 .yamllint
CONFIGURATION
No result to commande : ansible-config dump --only-changed
OS / ENVIRONMENT
Linux XXXX 5.13.0-41-generic #46~20.04.1-Ubuntu
STEPS TO REPRODUCE
My parameters :
kubeconfig :
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tL.....Qo=
server: https://48F2BD40CC5BBFCECD8DC24F8DXXXXXX.yl4.eu-west-3.eks.amazonaws.com
name: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
contexts:
- context:
cluster: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
namespace: test
user: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
name: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
current-context: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
kind: Config
preferences: {}
users:
- name: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- eu-west-3
- eks
- get-token
- --cluster-name
- aws-common-development
command: aws
env:
- name: AWS_PROFILE
value: common
interactiveMode: IfAvailable
provideClusterInfo: false
Playbook of test
- name: "Test k8s commande"
kubernetes.core.k8s_exec:
kubeconfig: "{{kubeconfig}}"
username: "{{kubectl_username}}"
context: "{{kubectl_context}}"
namespace: test
pod: network-tools
command: "ip a"
With parameters :
kubeconfig: "/home/<my-user>/.kube/config"
kubectl_context: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
kubectl_username: arn:aws:eks:eu-west-3:XXXXXXXXXXXX:cluster/aws-common-development
kubectl_namespace: test
EXPECTED RESULTS
I would like the command to run on the pod and return the result
ACTUAL RESULTS
Error message on ansible
ASK [test : Test k8s commande] ***************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to get client due to 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'f359f293-3066-48d5-997f-7d2a34563c43', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '733e9229-3fcc-40db-b964-912b55e96e9e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '2ae196cc-4de5-4a2d-aac5-431bb93d0b76', 'Date': 'Fri, 13 May 2022 14:55:13 GMT', 'Content-Length': '189'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"forbidden: User \\\\\"system:anonymous\\\\\" cannot get path \\\\\"/apis\\\\\"\",\"reason\":\"Forbidden\",\"details\":{},\"code\":403}\\n'\nOriginal traceback: \n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/dynamic/client.py\", line 55, in inner\n resp = func(self, *args, **kwargs)\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/dynamic/client.py\", line 270, in request\n api_response = self.client.call_api(\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py\", line 348, in call_api\n return self.__call_api(resource_path, method,\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py\", line 180, in __call_api\n response_data = self.request(\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py\", line 373, in request\n return self.rest_client.GET(url,\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py\", line 240, in GET\n return self.request(\"GET\", url,\n\n File \"/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py\", line 234, in request\n raise ApiException(http_resp=r)\n"}
@olahouze Are you able to successfully use kubectl to run the command using the same kubeconfig?
I am not able to reproduce this on my end. Are you able to successfully retrieve your ExecCredential with your exec based auth command? In your case, it should be:
$ AWS_PROFILE=common aws --region eu-west-3 eks get-token --cluster-name aws-common-development
I am having these same issues, and its VERY hard to reproduce. I am using a service account, so my kubeconfig is token based.
In random namespaces, I encounter random issues, even though permission are set accordingly. In other namespaces, things are perfectly fine. Here is an example:
{"changed": false, "msg": "Scale request failed: 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '253def2c-33e7-4be1-9ad1-d317de42297e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '03379d20-3ead-4315-a979-5544ad552e44', 'X-Kubernetes-Pf-Prioritylevel-Uid': '95596f23-5203-4fdd-babf-ac3ccd0dc14c', 'Date': 'Thu, 04 Aug 2022 14:42:48 GMT', 'Content-Length': '376'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"deployments.apps \\\\\"processor\\\\\" is forbidden: User \\\\\"system:serviceaccount:default:deployer\\\\\" cannot patch resource \\\\\"deployments/scale\\\\\" in API group \\\\\"apps\\\\\" in the namespace \\\\\"namespace\\\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"processor\",\"group\":\"apps\",\"kind\":\"deployments\"},\"code\":403}
Thing is, I use terraform which sets the permissions for everyone namespace the SAME for the deployer user. It is impossible that in one namespace it works, in the next namespace I can't get, and in a third namespace I cant patch the scale.
Also, using kubectl - it works just fine. Using ansible via the 'k8s_scale'? Fails randomly.
- name: scale processor up
kubernetes.core.k8s_scale:
api_version: v1
kind: Deployment
name: processor
namespace: "{{ namespace }}"
current_replicas: 0
replicas: 1
wait: false
delegate_to: 127.0.0.1
We need some additional information in trying to reproduce this.
- What version of Ansible core are you running? (2.9.6 is rather old at this point, 2.13.2 is the latest stable.)
- What version of kubernetes.core are you using? (2.3.2 is the latest stable.)
Also, community.kubernetes has been deprecated for about 2 years and has be removed from the latest ansible community distros. It shouldn't be in conflict with kubernetes.core, but it's worth eliminating.
ansible-playbook [core 2.11.6]
config file = None
python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
jinja version = 2.10.1
libyaml = True
if the kubernetes.core is in the ansible_collections then the version is 1.2.1 based on the change log. Let me know if there is a better way to check this.
@jbkc85 You can use ansible-galaxy collection list kubernetes.core to get the version.
I am encountering a similar problem when trying to use k8s_exec, NOT on EKS though:
I am using a ServiceAccount token to access the api, provided in a kubeconfig file. Using this same file, I am able to create a pod in the same namespace.
My exec task
- name: create target database in db cluster
kubernetes.core.k8s_exec:
kubeconfig: /home/user/migration/kubeconfig
namespace: '{{ migration_namespace }}'
pod: 'db-client-customer-st-{{ customer_id }}'
command: 'mysql -hcluster1-pxc -uroot -p$MYSQL_DB_PASSWORD -e "CREATE DATABASE {{ db-name }}"'
generates the following error (using -vvvv)
The full traceback is:
File "/tmp/ansible_kubernetes.core.k8s_exec_payload_ukpfy3hi/ansible_kubernetes.core.k8s_exec_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_exec.py", line 193, in execute_module
File "/usr/local/lib/python3.7/dist-packages/kubernetes/stream/stream.py", line 35, in _websocket_request
return api_method(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/kubernetes/client/api/core_v1_api.py", line 994, in connect_get_namespaced_pod_exec
return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.7/dist-packages/kubernetes/client/api/core_v1_api.py", line 1115, in connect_get_namespaced_pod_exec_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.7/dist-packages/kubernetes/client/api_client.py", line 353, in call_api
_preload_content, _request_timeout, _host)
File "/usr/local/lib/python3.7/dist-packages/kubernetes/client/api_client.py", line 184, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.7/dist-packages/kubernetes/stream/ws_client.py", line 525, in websocket_call
raise ApiException(status=0, reason=str(e))
fatal: [migration-test.ftapi.com]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_key": null,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"command": "mysql -hcluster1-pxc -uroot -p$MYSQL_DB_PASSWORD -e \"CREATE DATABASE customer_st_123456789\"",
"container": null,
"context": null,
"host": null,
"impersonate_groups": null,
"impersonate_user": null,
"kubeconfig": "/home/andy/migration/kubeconfig",
"namespace": "migration-vm-to-cluster",
"no_proxy": null,
"password": null,
"persist_config": null,
"pod": "db-client-customer-st-123456789",
"proxy": null,
"proxy_headers": null,
"username": null,
"validate_certs": null
}
},
"msg": "Failed to execute on pod db-client-customer-st-123456789 due to : (0)\nReason: Handshake status 403 Forbidden\n"
}
Ansible & Module versions I use:
~/ansible-playbook --version
ansible-playbook 2.10.8
config file = None
configured module search path = ['/home/<user>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
~/ ansible-galaxy collection list kubernetes.core
# /home/<user>/.ansible/collections/ansible_collections
Collection Version
--------------- -------
kubernetes.core 2.3.2
The k8s cluster I use is version 1.21
If I manually run kubectl exec on my terminal using the same kubeconfig file, I do not have any issues.
Hello.
Is there news on this case ?
I am always facing the problem
Best regards