smart_open.s3.iter_bucket fails with 403 unauthorised when using moto

[orionmoana]

trafficstars

Problem description

I'm attempting to implement tests for an application using smart_open.s3.iter_bucket to read files from an S3 bucket. In the tests I am using moto to mock S3 (in a similar way to the smart_open tests themselves). However when iter_bucket calls bucket.download_fileobj(key_name, buf) it is returning 403 unauthorised. I am able to make this call myself from within the tests without issue.

Steps/code to reproduce the problem

Example test that produces this issue on my system (based on smart_open tests):

@moto.mock_s3
class IterBucketTest(unittest.TestCase):

    def setUp(self):
        ignore_resource_warnings()

    def tearDown(self):
        cleanup_bucket()

    def test_iter_bucket(self):
        populate_bucket()
        results = list(smart_open.s3.iter_bucket(BUCKET_NAME))
        self.assertEqual(len(results), 10)

Result of running test_iter_bucket:

test_iter_bucket (common_test.smart_open_test.IterBucketTest) ... ERROR
NoneType: None

======================================================================
ERROR: test_iter_bucket (common_test.smart_open_test.IterBucketTest)
----------------------------------------------------------------------
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1254, in _download_key
    content_bytes = _download_fileobj(bucket, key_name)
  File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1271, in _download_fileobj
    bucket.download_fileobj(key_name, buf)
  File "/project/path/venv/lib/python3.9/site-packages/boto3/s3/inject.py", line 718, in bucket_download_fileobj
    return self.meta.client.download_fileobj(
  File "/project/path/venv/lib/python3.9/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/project/path/venv/lib/python3.9/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/project/path/venv/lib/python3.9/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/project/path/venv/lib/python3.9/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/project/path/venv/lib/python3.9/site-packages/s3transfer/download.py", line 340, in _submit
    response = client.head_object(
  File "/project/path/venv/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/project/path/venv/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/path/venv/lib/python3.9/site-packages/moto/core/models.py", line 111, in wrapper
    result = func(*args, **kwargs)
  File "/project/path/test/common_test/smart_open_test.py", line 49, in test_iter_bucket
    results = list(smart_open.s3.iter_bucket(BUCKET_NAME))
  File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1192, in iter_bucket
    for key_no, (key, content) in enumerate(result_iterator):
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

----------------------------------------------------------------------
Ran 1 test in 4.255s

FAILED (errors=1)

Versions

macOS-11.4-x86_64-i386-64bit Python 3.9.2 (default, Feb 24 2021, 13:26:09) [Clang 12.0.0 (clang-1200.0.32.29)] smart_open 5.1.0 boto3 1.17.93 moto 2.2.1

[piskvorky]

Might be related to limitations of moto which reportedly fails to propagate keys to forked processes: https://github.com/RaRe-Technologies/smart_open/pull/635#issue-702702227

[judahrand]

Might be related to limitations of moto which reportedly fails to propagate keys to forked processes: #635 (comment)

I succeeds to propagate keys to FORKED processes. The issue is that fork is not used by default (and actually breaks on newer MacOS versions) on Windows or MacOS. Instead SPAWN is used and this fails to copy the whole process memory space.