400 Error (InvalidArgument <- BadRequest) when uploading blobs to S3 compatible bucket with SigV4 from rake task

[AJLfleos]

Issue

I was trying to integrate CF cloud controller with a S3 compatible object storage solution. There are no issues when directly testing the fog-aws lib or from the cloud_controller_ng's fog client using v4 signature, but there are odd 400 error (Invalid Argument <- Bad Request) from the rake task buildpacks.rake

Context

With the same configuration,

1. everything works properly from testing the fog-aws lib directly: https://gist.github.com/AJLfleos/2a4144c57877b02a2721cda5bb35d16e

2. everything works properly from testing the fog client within the cloud_controller_ng: https://gist.github.com/AJLfleos/3d0ab93d342a7cb9bf58a546cdeccb46

3. testing blob upload from buildpacks.rake in cloud_contoller_ng failed with 400 error: https://gist.github.com/AJLfleos/f7f73d04da7cddffc33e8a74bf15232b

using the same configuration for fog below:

value: &blobstore-properties
    provider: AWS
    aws_access_key_id: "3YOH7HSNJSEOUZMGNOQ7"
    aws_secret_access_key: "kLyF4GKxi8qk7fQIaJrvW6ep9FoxzObn1aZBPkKK"
    region: "test_region"
    endpoint: "https://objectstorage.test_region.test"
    path_style: true
    aws_signature_version: 4
    connection_options:
      ssl_verify_peer: false

---

I tried to investigate what's going on thinking there might be something added from the rake task in buildpacks.rake but I didn't find anything insightful that can suggest the root-cause of the error.

Also, what's interesting, when I read the Cloud Foundry doc for using the S3 compatible bucket on blobstore, it uses aws_signature_version: '2' (https://docs.cloudfoundry.org/deploying/common/cc-blobstore-config.html section "Fog with Other S3 Compatible Stores"). I tried to use V2 signature, the upload seems to work but download failed which is probably due to the S3 compatible bucket I used doesn't support V2 on 'get' operation.

---

I need your help to provide insights what's wrong with using V4 signature on uploading blobs to S3 compatible bucket from the cloud_contoller_ng's buildpacks.rake that uses fog client invoking fog-aws lib.

[AJLfleos]

Could anyone from the community help?