Investigate why some Ceph object lock tests can't delete bucket at the end

[halkyon]

It appears some Ceph object lock tests fail to clean up the test bucket, because we return a "bucket not empty" error:

• https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_compliance/
• https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_governance_with_bypass/
• https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_governance_without_bypass/

The three failing tests are "changing mode" and at the end don't expicitly call DeleteObject to clean up the object, where other tests are doing this and passing correctly. Instead, these changing mode tests seem to be relying on the test runner to try and clean things up before it deletes the bucket. See nuke_bucket function here: https://github.com/ceph/s3-tests/blob/d245097771698774d36a0283e9ef69bf67f3ba76/s3tests_boto3/functional/init.py#L98.

We should investigate if there's something wrong here, like it couldn't delete some of the leftover object versions. It seems like it's supposed to wait until the retention period has expired before cleaning things up, but it wasn't clear by the trace in the test if it actually waited or not.

[storj-gerrit[bot]]

Change testsuite/ceph-s3-tests: partially re-enable changing mode tests mentions this issue.

[halkyon]

The current theory is retention times sent to S3 are truncated to seconds, but we might not be doing that. We might try time.Truncate(time.Second) on the requested retention expiration time to see if it helps.