Can you add support to cos of tencent cloud?

[james-li]

When I setup a gitlab server in tencent cloud and want to use the cos(object storage) to store upload artificate. COS uses a different api for put object.

PUT /<ObjectKey> HTTP/1.1
Host: <BucketName-APPID>.cos.<Region>.myqcloud.com
Date: GMT Date
x-cos-copy-source: <SourceBucketName-SourceAPPID>.cos.<SourceRegion>.myqcloud.com/<SourceObjectKey>
Content-Length: 0
Authorization: Auth String

So I have to modify fog-aws-3.14.0/lib/fog/aws/storage.rb, add following to work around.

diff --git a/lib/fog/aws/storage.rb b/lib/fog/aws/storage.rb
index 73e72aef1..6603ea2f7 100644
--- a/lib/fog/aws/storage.rb
+++ b/lib/fog/aws/storage.rb
@@ -629,7 +627,9 @@ module Fog
           host = params.delete(:host)
           port = params.delete(:port) || DEFAULT_SCHEME_PORT[scheme]
           params[:headers]['Host'] = host
-
+          if host.downcase =~ (/.*myqcloud.com.*/) && params[:headers]["x-amz-copy-source"] != nil
+            params[:headers]["x-amz-copy-source"] = "#{host}#{params[:headers]["x-amz-copy-source"]}"
+          end

           if @signature_version == 4
             params[:headers]['x-amz-date'] = date.to_iso8601_basic

It works but is very ugly. :(

[geemus]

Hey, thanks for providing a detailed report. I'm not personally very familiar with cos, so kind of going off what you've laid out here.

Could you provide an example of what methods/classes you are using to make the calls as well? I'm trying to figure out where the best place to tweak things might be and understanding the usage pattern will simplify that I think. Thanks!

[james-li]

The exception stack in gitlab is :

config/initializers/carrierwave_patch.rb:29:in `copy_to'
app/uploaders/object_storage.rb:393:in `store!'
app/services/upload_service.rb:18:in `execute'
app/controllers/concerns/uploads_actions.rb:17:in `create'
app/controllers/application_controller.rb:527:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:518:in `set_session_storage'
..

And copy_to call copy_object in lib/fog/aws/requests/storage/copy_object.rb

I am not familiar with ruby and can not find how to set host in this function. So I modify the function in storage.rb.

Just FYI

[geemus]

Thanks, I wanted to make sure I understood if you were making the calls directly (in which case there are perhaps easier fixes), or if it was being done on your behalf (which makes it harder, as you point out with your patch). I'm going to review some of the related docs and code to see what seems like the best approach here and will comment when I've got something.

[geemus]

Could you also share the contents of the carrierwave_patch file? I've been digging through fog-aws, carrierwave, and gitlab files but don't see the actual copy calls (so guessing by process of elimination they must be in the initializer). Thanks!

[james-li]

config/initializers/carrierwave_patch.rb

https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/carrierwave_patch.rb

[github-actions[bot]]

This issue has been marked inactive and will be closed if no further activity occurs.