HTTP Error 403: rate limit exceeded - az aks install cli

[bkozub]

Describe the bug Sometimes when using bitbucket pipelines from mcr.microsoft.com/azure-cli:latest image i get the 403 Error while az aks install cli command is executed

To Reproduce

occurs randomly

Expected behavior aks cli is installed

Environment summary

Additional context

Downloading client to "/usr/local/bin/kubectl" from "https://storage.googleapis.com/kubernetes-release/release/v1.19.4/bin/linux/amd64/kubectl"
Please ensure that /usr/local/bin is in your search PATH, so the `kubectl` command can be found.
CLIInternalError: The command failed with an unexpected error. Here is the traceback:
HTTP Error 403: rate limit exceeded
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 654, in execute
    raise ex
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 711, in _run_job
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 688, in _run_job
    result = cmd_copy(params)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 325, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 784, in default_command_handler
    return op(**command_args)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 329, in k8s_install_cli
    k8s_install_kubelogin(cmd, kubelogin_version, kubelogin_install_location)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 407, in k8s_install_kubelogin
    latest_release = urlopen(latest_release_url, context=context).read()
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded```

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm.

Issue Details

Description:

Describe the bug Sometimes when using bitbucket pipelines from mcr.microsoft.com/azure-cli:latest image i get the 403 Error while az aks install cli command is executed To Reproduce occurs randomly Expected behavior aks cli is installed Environment summary Additional context Downloading client to "/usr/local/bin/kubectl" from "https://storage.googleapis.com/kubernetes-release/release/v1.19.4/bin/linux/amd64/kubectl" Please ensure that /usr/local/bin is in your search PATH, so the `kubectl` command can be found. CLIInternalError: The command failed with an unexpected error. Here is the traceback: HTTP Error 403: rate limit exceeded Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 215, in invoke cmd_result = self.invocation.execute(args) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 654, in execute raise ex File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 711, in _run_job six.reraise(*sys.exc_info()) File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 688, in _run_job result = cmd_copy(params) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 325, in __call__ return self.handler(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 784, in default_command_handler return op(**command_args) File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 329, in k8s_install_cli k8s_install_kubelogin(cmd, kubelogin_version, kubelogin_install_location) File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 407, in k8s_install_kubelogin latest_release = urlopen(latest_release_url, context=context).read() File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen return opener.open(url, data, timeout) File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(*args) File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(*args) File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: rate limit exceeded``` </td> </tr> <tr> <th align="left">Author:</th> <td>bkozub</td> </tr> <tr> <th align="left">Assignees:</th> <td>-</td> </tr> <tr> <th align="left">Labels:</th> <td> `AKS`, `Service Attention` </td> </tr> <tr> <th align="left">Milestone:</th> <td>-</td> </tr> </table> </details>

[yonzhan]

aks

[waldekmanual]

any new about this?

[henrykironde]

I am getting the same error https://github.com/weecology/DeepForest-pytorch/pull/41/checks?check_run_id=2179945044

[kemmemha]

Is there any update on this issue? We are also encountering this issue pretty regularely. It appears in CI / CD after 50 installations "az aks install-cli". We use --base-src-url and -l over an Artifactory server with the hope to prevent this from happening but we same error occurs.

The issue seems to appear because kubelogin (even if used with -l parameter) is calling the Github API if it downloads the latest version: https://github.com/Azure/azure-cli/blob/edb293b58607450c64b634935c094405c995c212/src/azure-cli/azure/cli/command_modules/acs/custom.py#L438

so there is still 1 call to the Github API which will (if used too many times) exceed the rate limit.

[phenixita]

Hi team, any news on this? I'm working with a customer and we're facing the same issue.

[phenixita]

I suppose the limitation is the rate limit on Google servers, from where the cli downloads the tools. Source code

[stan-sz]

See also #19965

[timbodeit]

Suggested solution: Rather than increasing rate limits by IP, provide a second variant of the azure-cli Docker container, that already comes with the aks cli preinstalled. That way, pipelines based on Microsoft's Docker images wouldn't need to manually install it manually on each run in the first place.

@bkozub Would that resolve the issue for you as well?

Rel: https://github.com/Azure/azure-cli/issues/19965#issuecomment-1688162845

[hgebrael]

Any updates on this? I agree with @timbodeit. Why don't we have a second image that has the aks cli preinstalled?

[ekbaramundi]

Any new updates on this case? We still encounter this issue!

[fjbejarano-nn]

Is there any update on this issue? We are also encountering this issue pretty regularely. It appears in CI / CD after 50 installations "az aks install-cli". We use --base-src-url and -l over an Artifactory server with the hope to prevent this from happening but we same error occurs.

The issue seems to appear because kubelogin (even if used with -l parameter) is calling the Github API if it downloads the latest version:

https://github.com/Azure/azure-cli/blob/edb293b58607450c64b634935c094405c995c212/src/azure-cli/azure/cli/command_modules/acs/custom.py#L438

so there is still 1 call to the Github API which will (if used too many times) exceed the rate limit.

I have the same problem with recent installation. In hashicorp packer, when you installed pluging, this issue happened but they implemented a variable with a github token. You can now execute the installation with a parameter with that github token and the rate is incremented drasticaly and all plugins are installed but I don't know if we can use a param with a github token in the installation of az aks install-cli.

[mymichu]

We noticed the same behaviour/issue in our build infrastructure where we build docker images. Is there any update on this issue?

[safronau]

Any update?