fix: improve S3 API compatibility for ListObjects operations

[overtrue]

This PR fixes multiple S3 API compatibility issues, improving rustfs's compliance with AWS S3 API specifications. These fixes primarily address implementation details of ListObjects, ListObjectsV2, and ListObjectVersions APIs.

Type of Change

• [ ] New Feature
• [ ] Bug Fix
• [ ] Documentation
• [ ] Performance Improvement
• [x] Test/CI
• [ ] Refactor
• [ ] Other:

Related Issues

Summary of Changes

Changes

1. ListObjectsV2 KeyCount Calculation Fix

• Issue: The KeyCount field only counted Contents items, excluding CommonPrefixes
• Fix: Updated KeyCount calculation logic to include both Contents and CommonPrefixes counts
• Impact: Fixed test_bucket_listv2_delimiter_basic test case

2. ListObjectsV2 URL Encoding Support

• Issue: When EncodingType='url' was specified, Keys and Prefixes in the response were not URL-encoded
• Fix: Implemented URL encoding logic that preserves path separators (/) while encoding special characters (e.g., +, spaces)
• Impact: Fixed test_bucket_listv2_encoding_basic test case

3. ListObjectVersions NextVersionIdMarker Fix

• Issue: When NextVersionIdMarker was empty, the response included None values, causing boto3 parameter validation errors
• Fix: Filter out empty values, only include NextVersionIdMarker in the response when it has a value
• Impact: Eliminated 794 VersionIdMarker parameter validation errors

4. ListObjects NextMarker Calculation Fix

• Issue: The NextMarker field returned None, but should return the key of the last returned item
• Fix: When both Contents and CommonPrefixes exist, return the lexicographically greatest item; otherwise return the last key or prefix
• Impact: Fixed test_bucket_list_delimiter_prefix and test_bucket_list_delimiter_prefix_underscore test cases

Testing

Verified fixes using the ceph/s3-tests test suite:

• ✅ test_bucket_listv2_delimiter_basic - PASSED
• ✅ test_bucket_listv2_encoding_basic - PASSED
• ✅ test_bucket_list_delimiter_prefix - PASSED
• ✅ test_bucket_list_delimiter_prefix_underscore - PASSED
• ✅ Reduced VersionIdMarker parameter validation errors from 794 to 0

Code Quality

• ✅ All code passes cargo fmt formatting checks
• ✅ All code passes cargo clippy static analysis
• ✅ Added corresponding unit tests for each fix

Related Commits

• 044eb242 - fix: include common prefixes in ListObjectsV2 KeyCount
• c18d31dd - fix: implement URL encoding for ListObjectsV2 EncodingType parameter
• 87ea80e0 - fix: filter empty next_version_id_marker in ListObjectVersions
• 4f9bc833 - fix: calculate NextMarker correctly for ListObjects API
• 73c38f83 - fix: clean up whitespace in NextMarker calculation for ListObjects API

References

• AWS S3 ListObjectsV2 API Documentation
• AWS S3 ListObjects API Documentation
• AWS S3 ListObjectVersions API Documentation

Checklist

• [x] I have read and followed the CONTRIBUTING.md guidelines
• [x] Passed make pre-commit
• [x] Added/updated necessary tests
• [ ] Documentation updated (if needed)
• [x] CI/CD passed (if applicable)

Impact

• [ ] Breaking change (compatibility)
• [ ] Requires doc/config/deployment update
• [ ] Other impact:

Additional Notes

---

Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.