BadHostKeyException when attempting az acs kubernetes get-credentials

[jungho]

trafficstars

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes 1.8.1 Canadacentral

az --version azure-cli (2.0.31) python -V Python 2.7.14 python3 -V Python 3.6.5

What happened:

Successfully created cluster, when trying to get the kubeconfig credentials, the parimiko library raises a BadHostKeyException.

Created cluster with:

az group create -n acs-cluster -l canadacentral az acs create -n acs-cluster -g acs-cluster -t Kubernetes --master-count 1 --agent-count 2 --orchestrator-version 1.8.1 --generate-ssh-keys

Get credentials with:

az acs kubernetes get-credentials -n acs-cluster -g acs-cluster --debug

I get the following output (modified to remove subscription info)

msrest.pipeline : Configuring request: timeout=100, verify=True, cert=None
msrest.pipeline : Configuring redirects: allow=True, max=30
msrest.pipeline : Configuring proxies: ''
msrest.pipeline : Evaluate proxies against ENV settings: True
msrest.pipeline : Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
urllib3.connectionpool : Starting new HTTPS connection (1): management.azure.com
urllib3.connectionpool : https://management.azure.com:443 "GET /subscriptions/xxx/resourceGroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster ?api-version=2017-07-01 HTTP/1.1" 200 None
msrest.http_logger : Request URL: 'https://management.azure.com/subscriptions/REMOVED/resourceGroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster ?api-version=2017-07-01'
msrest.http_logger : Request method: 'GET'
msrest.http_logger : Request headers:
msrest.http_logger : 'Connection': 'keep-alive'
msrest.http_logger : 'Accept-Encoding': 'gzip, deflate'
msrest.http_logger : 'Accept': 'application/json'
msrest.http_logger : 'User-Agent': 'python/2.7.14 (Linux-4.15.14-300.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven) requests/2.18.4 msrest/0.4.27 msrest_azure/0.4.25 azure-mgmt-containerservice/3.0.1 Azu re-SDK-For-Python AZURECLI/2.0.31'
msrest.http_logger : 'Authorization': '*****'
msrest.http_logger : 'x-ms-client-request-id': 'dfdafds'
msrest.http_logger : 'CommandName': 'acs kubernetes get-credentials'
msrest.http_logger : 'Content-Type': 'application/json; charset=utf-8'
msrest.http_logger : 'accept-language': 'en-US'
msrest.http_logger : Request body:
msrest.http_logger : None
msrest.http_logger : Response status: 200
msrest.http_logger : Response headers:
msrest.http_logger : 'Cache-Control': 'no-cache'
msrest.http_logger : 'Pragma': 'no-cache'
msrest.http_logger : 'Transfer-Encoding': 'chunked'
msrest.http_logger : 'Content-Type': 'application/json'
msrest.http_logger : 'Content-Encoding': 'gzip'
msrest.http_logger : 'Expires': '-1'
msrest.http_logger : 'Vary': 'Accept-Encoding'
msrest.http_logger : 'x-ms-correlation-request-id': '2aa92282-6bd7-4763-973d-1a4dd60582f7'
msrest.http_logger : 'x-ms-request-id': '16259863-8bac-40a4-960e-de768ff482f3'
msrest.http_logger : 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
msrest.http_logger : 'Server': 'nginx'
msrest.http_logger : 'x-ms-ratelimit-remaining-subscription-reads': '14992' msrest.http_logger : 'x-ms-routing-request-id': 'CANADACENTRAL:20180415T122044Z:2aa92282-6bd7-4763-973d-1a4dd60582f7' msrest.http_logger : 'X-Content-Type-Options': 'nosniff' msrest.http_logger : 'Date': 'Sun, 15 Apr 2018 12:20:44 GMT' msrest.http_logger : Response content: msrest.http_logger : { "id": "/subscriptions/REMOVED/resourcegroups/acs-cluster/providers/Microsoft.ContainerService/containerServices/acs-cluster", "location": "canadacentral", "name": "acs-cluster", "type": "Microsoft.ContainerService/ContainerServices", "properties": { "provisioningState": "Succeeded", "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.8.1" }, "masterProfile": { "count": 1, "dnsPrefix": "acs-cluste-acs-cluster-17ef54mgmt", "vmSize": "Standard_D2_v2", "firstConsecutiveStaticIP": "10.240.255.5", "storageProfile": "ManagedDisks", "fqdn": "acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com" }, "agentPoolProfiles": [ { "name": "agentpool0", "count": 2, "vmSize": "Standard_D2_v2", "dnsPrefix": "", "fqdn": "", "storageProfile": "StorageAccount", "osType": "Linux" } ], "linuxProfile": { "adminUsername": "azureuser", "ssh": { "publicKeys": [ { "keyData": "ssh-rsa key REMOVED" } ] } }, "servicePrincipalProfile": { "clientId": "REMOVED" } } } paramiko.transport : starting thread (client mode): 0x886dd3d0L paramiko.transport : Local version/idstring: SSH-2.0-paramiko_2.4.1 paramiko.transport : Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2) paramiko.transport : kex algos:[u'[email protected]', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] s erver key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'[email protected]', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@ope nssh.com', u'[email protected]'] server encrypt:[u'[email protected]', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'[email protected]', u'[email protected]'] client mac:[u'umac-64 [email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'hmac-sha2-2 56', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'[email protected]', u'um [email protected]', u'[email protected]', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'[email protected]'] server compress:[u'none', u'[email protected]'] client lang:[u''] s erver lang:[u''] kex follows?False paramiko.transport : Kex agreed: ecdh-sha2-nistp256 paramiko.transport : HostKey agreed: ecdsa-sha2-nistp256 paramiko.transport : Cipher agreed: aes128-ctr paramiko.transport : MAC agreed: hmac-sha2-256 paramiko.transport : Compression agreed: none paramiko.transport : kex engine KexNistp256 specified hash_algo paramiko.transport : Switch to new keys ... ('acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com', <paramiko.ecdsakey.ECDSAKey object at 0x7f99886d5c90>, <paramiko.ecdsakey.ECDSAKey object at 0x7f998b6a3850>)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 347, in execute
six.reraise(*sys.exc_info())
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 319, in execute
result = cmd(params)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/init.py", line 180, in call
return super(AzCliCommand, self).call(*args, **kwargs)
File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 109, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/init.py", line 420, in default_command_handler
result = op(**command_args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 926, in k8s_get_credentials
_k8s_get_credentials_internal(name, acs_info, path, ssh_key_file)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/custom.py", line 947, in _k8s_get_credentials_internal
'.kube/config', path_candidate, key_filename=ssh_key_file)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acs/acs_client.py", line 72, in secure_copy
ssh.connect(host, username=user, pkey=pkey, sock=proxy)
File "/usr/lib64/az/lib/python2.7/site-packages/paramiko/client.py", line 409, in connect
raise BadHostKeyException(hostname, server_key, our_key)
BadHostKeyException: ('acs-cluste-acs-cluster-17ef54mgmt.canadacentral.cloudapp.azure.com', <paramiko.ecdsakey.ECDSAKey object at 0x7f99886d5c90>, <paramiko.ecdsakey.ECDSAKey object at 0x7f998b6a3850>)
paramiko.transport : EOF in transport thread

What you expected to happen:

To get credentials to access the cluster using kubectl.

How to reproduce it (as minimally and precisely as possible):

Follow the commands above.

Anything else we need to know:

I have seen this issue before, sometimes, when I downgraded the az client, it seemed to correct the issue. This time, it did not. I have also tried different regions (canadaeast) and in the past, it corrected the issue, this time it did not. I have also tried creating the cluster by specifying my own key then getting the credentials with --ssh-key-value option. Same result.

[jungho]

Note, this issue does not exist in Canada East region. It does exist in Canada Central.