Fails checking the region ap-northeast-3

[varuzam]

seems that something was changed on AWS side because it worked a while ago.

awslimitchecker 11.0.0

Installation Method

virtualenv

Actual Output

~# awslimitchecker -r ap-northeast-3

/tmp/venv/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
awslimitchecker 11.0.0 is AGPL-licensed free software; all users have a right to the full source code of this version. See <https://github.com/jantman/awslimitchecker>
Traceback (most recent call last):
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/venv/lib/python3.6/site-packages/botocore/httpsession.py", line 323, in send
    chunked=self._chunked(request.headers),
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/tmp/venv/lib/python3.6/site-packages/urllib3/connection.py", line 177, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f04f23e0358>, 'Connection to ds.ap-northeast-3.amazonaws.com timed out. (connect timeout=60)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/venv/bin/awslimitchecker", line 11, in <module>
    sys.exit(console_entry_point())
  File "/tmp/venv/lib/python3.6/site-packages/awslimitchecker/runner.py", line 550, in console_entry_point
    r.console_entry_point()
  File "/tmp/venv/lib/python3.6/site-packages/awslimitchecker/runner.py", line 520, in console_entry_point
    res, problems, problem_str = self.check_thresholds(metrics)
  File "/tmp/venv/lib/python3.6/site-packages/awslimitchecker/runner.py", line 334, in check_thresholds
    service=self.service_name
  File "/tmp/venv/lib/python3.6/site-packages/awslimitchecker/checker.py", line 631, in check_thresholds
    cls._update_limits_from_api()
  File "/tmp/venv/lib/python3.6/site-packages/awslimitchecker/services/directoryservice.py", line 127, in _update_limits_from_api
    resp = self.conn.get_directory_limits()
  File "/tmp/venv/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/tmp/venv/lib/python3.6/site-packages/botocore/endpoint.py", line 256, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/tmp/venv/lib/python3.6/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/tmp/venv/lib/python3.6/site-packages/botocore/httpsession.py", line 347, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ds.ap-northeast-3.amazonaws.com/"

[robpickerill]

hey, I quickly checked and directory services isn't listed as a supported service for Osaka (ap-northeast-3): https://docs.amazonaws.cn/en_us/general/latest/gr/ds_region.html

Does it fix anything if you ran this with --skip-service for Directory Service?

[varuzam]

It works. thanks. But I assumed that awslimitchecker can determine by itself which services are available in a region.

[jantman]

Apologies for letting this sit so long.

Unfortunately, awslimitchecker can't currently figure out on its own which services are available in a region. The real problem here is that AWS seems to have deployed endpoint DNS for every service in every region. It used to be that if a service wasn't supported in a region, either the DNS (i.e. ds.ap-northeast-3.amazonaws.com) wouldn't resolve, which is handled by awslimitchecker, or the service would respond with a 503 or something, which is also handled. It now seems that AWS is serving DNS for every service in every region and just not having anything listening (hence connection timeouts) in regions where the service isn't supported.

There is now a solution for this built in to boto3; we can call boto3.session.Session().get_available_regions(self.api_name) and get back a list of regions where the service is supported, according to the AWS autogenerated API docs (the JSON API descriptions baked into botocore).

My concern with that, though, is that it will require a boto3/botocore update to recognize new regions for a service. And, for anyone using the Docker image, it would require a new awslimitchecker release... which has been very infrequent lately.

I think my gut reaction is to recommend just using --skip-service for now, unless someone can come up with a better solution that doesn't require an awslimitchecker release every time a service is enabled in a new region.