thumbor-s3-docker icon indicating copy to clipboard operation
thumbor-s3-docker copied to clipboard

Published 20 hours ago • verified publisher icon beeyev

empty endpoint URL breaks uploads

Open joe-at-startupmedia opened this issue 11 months ago • 4 comments

From what I understand, AWS_RESULT_STORAGE_S3_ENDPOINT_URL is an optional variable that should be specified when using custom s3 endpoint other than AWS. When I do not specify the value, your config template defaults it to None which throws the following error:

[thumbor-staging] [2024-03-26 02:27:47] 2024-03-26 02:27:47 thumbor:ERROR ERROR: Traceback (most recent call last):
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
[thumbor-staging] [2024-03-26 02:27:47]     result = await result
[thumbor-staging] [2024-03-26 02:27:47]              ^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/thumbor/handlers/upload.py", line 67, in post
[thumbor-staging] [2024-03-26 02:27:47]     await self.write_file(image_id, body)
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/thumbor/handlers/__init__.py", line 1084, in write_file
[thumbor-staging] [2024-03-26 02:27:47]     await storage.put(file_id, body)
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/thumbor_aws/storage.py", line 98, in put
[thumbor-staging] [2024-03-26 02:27:47]     path = await self.upload(
[thumbor-staging] [2024-03-26 02:27:47]            ^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/thumbor_aws/s3_client.py", line 110, in upload
[thumbor-staging] [2024-03-26 02:27:47]     async with self.get_client() as client:
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/session.py", line 25, in __aenter__
[thumbor-staging] [2024-03-26 02:27:47]     self._client = await self._coro
[thumbor-staging] [2024-03-26 02:27:47]                    ^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/session.py", line 215, in _create_client
[thumbor-staging] [2024-03-26 02:27:47]     client = await client_creator.create_client(
[thumbor-staging] [2024-03-26 02:27:47]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/client.py", line 77, in create_client
[thumbor-staging] [2024-03-26 02:27:47]     client_args = self._get_client_args(
[thumbor-staging] [2024-03-26 02:27:47]                   ^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/client.py", line 283, in _get_client_args
[thumbor-staging] [2024-03-26 02:27:47]     return args_creator.get_client_args(
[thumbor-staging] [2024-03-26 02:27:47]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/args.py", line 76, in get_client_args
[thumbor-staging] [2024-03-26 02:27:47]     endpoint = endpoint_creator.create_endpoint(
[thumbor-staging] [2024-03-26 02:27:47]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:27:47]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py", line 308, in create_endpoint
[thumbor-staging] [2024-03-26 02:27:47]     raise ValueError("Invalid endpoint: %s" % endpoint_url)
[thumbor-staging] [2024-03-26 02:27:47] ValueError: Invalid endpoint: None
[thumbor-staging] [2024-03-26 02:27:47] 
[thumbor-sta

joe-at-startupmedia avatar Mar 26 '24 02:03 joe-at-startupmedia

I've used the following environment variables:

ENV

UPLOAD_ENABLED=True
LOADER=tc_aws.loaders.s3_loader
STORAGE=tc_aws.storages.s3_storage
RESULT_STORAGE=tc_aws.result_storages.s3_storage
ALLOW_UNSAFE_URL=True
RESULT_STORAGE_STORES_UNSAFE=True
UPLOAD_PHOTO_STORAGE=tc_aws.storages.s3_storage
AWS_LOADER_REGION_NAME = 'us-east-1'
AWS_LOADER_BUCKET_NAME = 'thumbor-cdn.staging.company.com'
AWS_LOADER_S3_SECRET_ACCESS_KEY = 'qjXXXXXXXXXXXXXXXX4Sn'
AWS_LOADER_S3_ACCESS_KEY_ID = 'AKXXXXXXXXXXXXXXXELQ'
AWS_LOADER_ROOT_PATH = 'loader'
AWS_STORAGE_REGION_NAME = 'us-east-1'
AWS_STORAGE_BUCKET_NAME = 'thumbor-cdn.staging.company.com'
AWS_STORAGE_S3_SECRET_ACCESS_KEY = 'qjXXXXXXXXXXXXXXXX4Sn'
AWS_STORAGE_S3_ACCESS_KEY_ID = ''AKXXXXXXXXXXXXXXXELQ'
AWS_STORAGE_ROOT_PATH = 'storage'
AWS_RESULT_STORAGE_REGION_NAME = 'us-east-1'
AWS_RESULT_STORAGE_BUCKET_NAME = 'thumbor-cdn.staging.company.com'
AWS_RESULT_STORAGE_S3_SECRET_ACCESS_KEY = 'qjXXXXXXXXXXXXXXXX4Sn'
AWS_RESULT_STORAGE_S3_ACCESS_KEY_ID = 'AKXXXXXXXXXXXXXXXELQ'
AWS_RESULT_STORAGE_ROOT_PATH = 'result'

Error:

[thumbor-staging] [2024-03-26 02:27:47] ValueError: Invalid endpoint: None

Appending the endpoint variables to those above results in the following error:

ENV

AWS_LOADER_S3_ENDPOINT_URL = 'https://thumbor-cdn.staging.company.com'
AWS_STORAGE_S3_ENDPOINT_URL = 'https://thumbor-cdn.staging.company.com'
AWS_RESULT_STORAGE_S3_ENDPOINT_URL = 'https://thumbor-cdn.staging.company.com'

Error

[thumbor-staging] [2024-03-26 02:51:07] 2024-03-26 02:51:07 thumbor:ERROR Unable to upload image to storage/77f808481d4947e69e8432fca64f44ee: Could not connect to the endpoint URL: "https://thumbor-cdn.staging.company.com/thumbor-cdn.staging.company.com/storage/77f808481d4947e69e8432fca64f44ee" (<class 'botocore.exceptions.EndpointConnectionError'>)
[thumbor-staging] [2024-03-26 02:51:07] 2024-03-26 02:51:07 thumbor:ERROR ERROR: Traceback (most recent call last):
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1173, in _create_direct_connection
[thumbor-staging] [2024-03-26 02:51:07]     hosts = await asyncio.shield(host_resolved)
[thumbor-staging] [2024-03-26 02:51:07]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 884, in _resolve_host
[thumbor-staging] [2024-03-26 02:51:07]     addrs = await self._resolver.resolve(host, port, family=self._family)
[thumbor-staging] [2024-03-26 02:51:07]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/resolver.py", line 33, in resolve
[thumbor-staging] [2024-03-26 02:51:07]     infos = await self._loop.getaddrinfo(
[thumbor-staging] [2024-03-26 02:51:07]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 868, in getaddrinfo
[thumbor-staging] [2024-03-26 02:51:07]     return await self.run_in_executor(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
[thumbor-staging] [2024-03-26 02:51:07]     result = self.fn(*self.args, **self.kwargs)
[thumbor-staging] [2024-03-26 02:51:07]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
[thumbor-staging] [2024-03-26 02:51:07]     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
[thumbor-staging] [2024-03-26 02:51:07]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07] socket.gaierror: [Errno -2] Name does not resolve
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] The above exception was the direct cause of the following exception:
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] Traceback (most recent call last):
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/httpsession.py", line 208, in send
[thumbor-staging] [2024-03-26 02:51:07]     response = await self._session.request(
[thumbor-staging] [2024-03-26 02:51:07]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 578, in _request
[thumbor-staging] [2024-03-26 02:51:07]     conn = await self._connector.connect(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 544, in connect
[thumbor-staging] [2024-03-26 02:51:07]     proto = await self._create_connection(req, traces, timeout)
[thumbor-staging] [2024-03-26 02:51:07]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 911, in _create_connection
[thumbor-staging] [2024-03-26 02:51:07]     _, proto = await self._create_direct_connection(req, traces, timeout)
[thumbor-staging] [2024-03-26 02:51:07]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1187, in _create_direct_connection
[thumbor-staging] [2024-03-26 02:51:07]     raise ClientConnectorError(req.connection_key, exc) from exc
[thumbor-staging] [2024-03-26 02:51:07] aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host thumbor-cdn.staging.company.com:443 ssl:default [Name does not resolve]
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] During handling of the above exception, another exception occurred:
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] Traceback (most recent call last):
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/thumbor_aws/s3_client.py", line 122, in upload
[thumbor-staging] [2024-03-26 02:51:07]     response = await client.put_object(**settings)
[thumbor-staging] [2024-03-26 02:51:07]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/client.py", line 388, in _make_api_call
[thumbor-staging] [2024-03-26 02:51:07]     http, parsed_response = await self._make_request(
[thumbor-staging] [2024-03-26 02:51:07]                             ^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/client.py", line 416, in _make_request
[thumbor-staging] [2024-03-26 02:51:07]     return await self._endpoint.make_request(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py", line 100, in _send_request
[thumbor-staging] [2024-03-26 02:51:07]     while await self._needs_retry(
[thumbor-staging] [2024-03-26 02:51:07]           ^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py", line 262, in _needs_retry
[thumbor-staging] [2024-03-26 02:51:07]     responses = await self._event_emitter.emit(
[thumbor-staging] [2024-03-26 02:51:07]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/hooks.py", line 66, in _emit
[thumbor-staging] [2024-03-26 02:51:07]     response = await resolve_awaitable(handler(**kwargs))
[thumbor-staging] [2024-03-26 02:51:07]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
[thumbor-staging] [2024-03-26 02:51:07]     return await obj
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/retryhandler.py", line 107, in _call
[thumbor-staging] [2024-03-26 02:51:07]     if await resolve_awaitable(self._checker(**checker_kwargs)):
[thumbor-staging] [2024-03-26 02:51:07]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
[thumbor-staging] [2024-03-26 02:51:07]     return await obj
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/retryhandler.py", line 126, in _call
[thumbor-staging] [2024-03-26 02:51:07]     should_retry = await self._should_retry(
[thumbor-staging] [2024-03-26 02:51:07]                    ^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/retryhandler.py", line 165, in _should_retry
[thumbor-staging] [2024-03-26 02:51:07]     return await resolve_awaitable(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
[thumbor-staging] [2024-03-26 02:51:07]     return await obj
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/retryhandler.py", line 174, in _call
[thumbor-staging] [2024-03-26 02:51:07]     checker(attempt_number, response, caught_exception)
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/botocore/retryhandler.py", line 247, in __call__
[thumbor-staging] [2024-03-26 02:51:07]     return self._check_caught_exception(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
[thumbor-staging] [2024-03-26 02:51:07]     raise caught_exception
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py", line 181, in _do_get_response
[thumbor-staging] [2024-03-26 02:51:07]     http_response = await self._send(request)
[thumbor-staging] [2024-03-26 02:51:07]                     ^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py", line 285, in _send
[thumbor-staging] [2024-03-26 02:51:07]     return await self.http_session.send(request)
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/aiobotocore/httpsession.py", line 253, in send
[thumbor-staging] [2024-03-26 02:51:07]     raise EndpointConnectionError(endpoint_url=request.url, error=e)
[thumbor-staging] [2024-03-26 02:51:07] botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://thumbor-cdn.staging.company.com/thumbor-cdn.staging.company.com/storage/77f808481d4947e69e8432fca64f44ee"
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] During handling of the above exception, another exception occurred:
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] Traceback (most recent call last):
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
[thumbor-staging] [2024-03-26 02:51:07]     result = await result
[thumbor-staging] [2024-03-26 02:51:07]              ^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/thumbor/handlers/upload.py", line 67, in post
[thumbor-staging] [2024-03-26 02:51:07]     await self.write_file(image_id, body)
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/thumbor/handlers/__init__.py", line 1084, in write_file
[thumbor-staging] [2024-03-26 02:51:07]     await storage.put(file_id, body)
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/thumbor_aws/storage.py", line 98, in put
[thumbor-staging] [2024-03-26 02:51:07]     path = await self.upload(
[thumbor-staging] [2024-03-26 02:51:07]            ^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07]   File "/usr/local/lib/python3.11/site-packages/thumbor_aws/s3_client.py", line 126, in upload
[thumbor-staging] [2024-03-26 02:51:07]     raise RuntimeError(msg)  # pylint: disable=raise-missing-from
[thumbor-staging] [2024-03-26 02:51:07]     ^^^^^^^^^^^^^^^^^^^^^^^
[thumbor-staging] [2024-03-26 02:51:07] RuntimeError: Unable to upload image to storage/77f808481d4947e69e8432fca64f44ee: Could not connect to the endpoint URL: "https://thumbor-cdn.staging.company.com/thumbor-cdn.staging.company.com/storage/77f808481d4947e69e8432fca64f44ee" (<class 'botocore.exceptions.EndpointConnectionError'>)
[thumbor-staging] [2024-03-26 02:51:07] 
[thumbor-staging] [2024-03-26 02:51:07] 2024-03-26 02:51:07 tornado.access:ERROR 500 POST /image (172.71.150.178) 8986.47ms

It's obviously appending the endpoint URL to the bucketname which is not the desired end result.

joe-at-startupmedia avatar Mar 26 '24 02:03 joe-at-startupmedia

That's weird, because by default it's expected to be None - https://github.com/thumbor/thumbor-aws#result-storage

beeyev avatar Mar 30 '24 19:03 beeyev

That's weird, because by default it's expected to be None - https://github.com/thumbor/thumbor-aws#result-storage

My suspicion is that unlike True and False, None is parsed as a string. That's why all configuration variables set to None are commented out which results in them not being evaluated by the configuration parser (derpconf?).

Simply commenting out the variable is sufficient because the configuration module sets the value to None by default. The difference being None is evaluated as its python type and not casted to a string. https://github.com/thumbor/thumbor-aws/blob/f8e541c7bbc2b7a7a70489e0baf6710f4435511e/thumbor_aws/loader.py#L48

But I could be wrong. Can you find an example of a config properly evaluating a value of None when it's not commented out? All examples I've seen have been commented out.

joe-at-startupmedia avatar Mar 30 '24 20:03 joe-at-startupmedia

I checked once again, and indeed there is a problem. I believe there is a problem in the aws plugin itself. I suggest posting this issue there

beeyev avatar Mar 31 '24 20:03 beeyev