Cannot use S3 with GITLAB_ARTIFACTS_OBJECT_STORE

[ibaranov91]

There seem to be some issue with GITLAB_ARTIFACTS_OBJECT_STORE_* parameters. It seems to not be working correctly with S3 URLs.

We're using the following parameters

      - GITLAB_ARTIFACTS_OBJECT_STORE_ENABLED=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_REMOTE_DIRECTORY="sample-bucket-name"
      - GITLAB_ARTIFACTS_OBJECT_STORE_DIRECT_UPLOAD=false
      - GITLAB_ARTIFACTS_OBJECT_STORE_PROXY_DOWNLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_BACKGROUND_UPLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=false
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_PROVIDER=AWS
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID=******
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY=******
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_REGION=us-east-1
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST="s3.us-east-1.amazonaws.com"

In this case when trying to run rake gitlab:artifacts:migrate we'll get bad URI(is not URI?): "://sample-bucket-name.s3.amazonaws.com:"

This URL doesn't seem correct since it's missing a region, it's even more interesting because we specified GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST and in theory it should use it when forming URL, but it does not. We tried removing double quotes around GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST, we tried not using GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST at all, we tried using GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=true instead of false, none of these seem to give correct URI for the S3.

[ibaranov91]

Actually we made it work, seems that the issue is with the 'host' parameter. It's depreciated now in the gitlab. We tried to use the endpoint parameter instead, even though we do not use minio and it worked. So parameters we used are:

      - GITLAB_ARTIFACTS_OBJECT_STORE_ENABLED=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_REMOTE_DIRECTORY=sample-bucket-name
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_ENDPOINT=https://s3.us-east-1.amazonaws.com
      - GITLAB_ARTIFACTS_OBJECT_STORE_DIRECT_UPLOAD=false
      - GITLAB_ARTIFACTS_OBJECT_STORE_PROXY_DOWNLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_BACKGROUND_UPLOAD=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=true
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_PROVIDER=AWS
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID=***
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY=******
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_REGION=us-east-1
      - GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST=s3.us-east-1.amazonaws.com

I assume we don't actually need to specify GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_HOST and it's very important to use GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=true so the URL gets formed correctly with GITLAB_ARTIFACTS_OBJECT_STORE_CONNECTION_AWS_ENDPOINT