NSFS | S3 | Versioning: After creating 5 object versions in parallel threads ListObjectVersions only lists two versions

[hseipp]

Environment info

• NooBaa Version: 5.15.4
• Platform: CentOS 8, deployed using https://github.com/IBM/StorageScaleVagrant

Actual behavior

Five multiple threads perform PutObject to the same key in a versioned bucket. After the operations completed, ListObjectVersions only delivers two objects.

Expected behavior

PutObject in simultaneously running threads to the same key should lead to as many versions as threads got spawned.

Steps to reproduce

Run Ceph s3-tests, method test_versioned_concurrent_object_create_concurrent_remove():

S3TEST_CONF=s3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_versioned_concurrent_object_create_concurrent_remove
...
        for i in range(5):
            t = _do_create_versioned_obj_concurrent(client, bucket_name, key, num_versions)
            _do_wait_completion(t)
    
            response = client.list_object_versions(Bucket=bucket_name)
            versions = response['Versions']
    
>           assert len(versions) == num_versions
E           assert 2 == 5
E            +  where 2 = len([{'ETag': '"mtime-d3roiq22g5c0-ino-1c03"', 'IsLatest': True, 'Key': 'myobj', 'LastModified': datetime.datetime(2024, 8..., 'IsLatest': True, 'Key': 'myobj', 'LastModified': datetime.datetime(2024, 8, 28, 16, 18, 38, tzinfo=tzlocal()), ...}])

Please note that s3-tests outputs the following in the "warnings summary" for the operation:

s3tests_boto3/functional/test_s3.py::test_versioned_concurrent_object_create_concurrent_remove
  /home/vagrant/s3-tests/.tox/py/lib/python3.8/site-packages/_pytest/threadexception.py:82: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-2
  
  Traceback (most recent call last):
    File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File "/usr/lib64/python3.8/threading.py", line 870, in run
      self._target(*self._args, **self._kwargs)
    File "/home/vagrant/s3-tests/s3tests_boto3/functional/test_s3.py", line 8034, in _do_create_object
      client.put_object(Bucket=bucket_name, Key=key, Body=body)
    File "/home/vagrant/s3-tests/.tox/py/lib/python3.8/site-packages/botocore/client.py", line 569, in _api_call
      return self._make_api_call(operation_name, kwargs)
    File "/home/vagrant/s3-tests/.tox/py/lib/python3.8/site-packages/botocore/client.py", line 1023, in _make_api_call
      raise error_class(parsed_response, operation_name)
  botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the PutObject operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
  
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

which seems to be a wrong error message as we are only using one bucket name for all the operations and the bucket was created before PutObject got executed.

More information - Screenshots / Logs / Other output

Noobaa log file with "all" log level:

noobaa_20240828_1836.log.gz