Boto instrumentation incorrectly assumes structure of URLs

[brett-lempereur]

The boto OpBeat integration incorrectly makes some assumptions about the structure of URLs and fails when those assumptions don't hold. To replicate:

1. Enable storages support for the Boto3 backend.
2. Change the setting AWS_S3_ENDPOINT_URL to a hostname that does not match the service-region convention, more specifically one that does not contain a '.' character.
3. Attempt to perform an action (e.g, deleting an object from an S3 bucket) on the bucket.

This will cause an error when unpacking the hostname string, shown in the traceback below.

The error was discovered while running unit tests in a docker-compose environment that simulates our production deployments, with a local instance of the minio S3 server.

======================================================================
ERROR: test_delete (--REDACTED--)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "--REDACTED--", line 297, in test_delete
    response = self.client.delete(url)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 323, in delete
    path, data=data, format=format, content_type=content_type, **extra)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 224, in delete
    return self.generic('DELETE', path, data, content_type, **extra)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 237, in generic
    method, path, data, content_type, secure, **extra)
  File "/usr/local/lib/python3.6/site-packages/django/test/client.py", line 404, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 288, in request
    return super(APIClient, self).request(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 240, in request
    request = super(APIRequestFactory, self).request(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/test/client.py", line 485, in request
    raise exc_value
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/viewsets.py", line 95, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 491, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/mixins.py", line 93, in destroy
    self.perform_destroy(instance)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/mixins.py", line 97, in perform_destroy
    instance.delete()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 891, in delete
    return collector.delete()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/deletion.py", line 307, in delete
    sender=model, instance=obj, using=self.using
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/app/kubota/assets/storage.py", line 64, in delete_file
    instance.file.delete(save=False)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 105, in delete
    self.storage.delete(self.name)
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 471, in delete
    self.bucket.Object(self._encode_name(name)).delete()
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 63, in __call__
    args, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 222, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/botocore.py", line 21, in call
    service, region, _ = parsed_url.hostname.split('.', 2)
ValueError: not enough values to unpack (expected 3, got 1)

[beniwohli]

Thanks for the report @brett-lempereur! I don't have the setup to properly test this, but if I understand you correctly, https://github.com/opbeat/opbeat_python/pull/199 should fix this. Would you mind trying out that branch in your test suite?