Add support for using S3-compatible blob storage providers in Bitbucket Server and GHES <3.8 migrations

[timrogers]

When migrating from GitHub Enterprise Server <3.8 or Bitbucket Server/Data Center, the CLI uploads your migration archive to either AWS S3 or Azure Blob Storage.

Some GitHub customers use other blob storage services which are compatible with the AWS S3 API - common examples include MinIO and IBM Cloud Object Storage.

We should add support for these providers. Broadly speaking, this should just require an S3 API endpoint to be specified to override AWS's defaults, but we'd need to test that.

[obscurerichard]

I'm considering exploring this as part of an upcoming migration using gh-gei... see https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html for the AWS docs on this. Specifying AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3 as an environment variable should be enough to allow MinIO or another S3-API-compatible service to work.

cc: @jerem-uzoma

[timrogers]

I'm considering exploring this as part of an upcoming migration using gh-gei... see https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html for the AWS docs on this. Specifying AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3 as an environment variable should be enough to allow MinIO or another S3-API-compatible service to work.

cc: @jerem-uzoma

We'd definitely be very open to that contribution. I'd suggest that you try to get review as early as possible so we can provide feedback and maximise the changes of getting this merged.

Just to highlight for you and anyone else reading this issue, for GitHub Enterprise Server 3.8 onwards, the blob storage upload is performed by the server itself, so CLI changes won't be sufficient to add "S3 compatible storage" support for that case.

[obscurerichard]

I don't think that any changes will be required to get this to work, but that has yet to be confirmed by experimentation.

[timrogers]

That's a good point. It may well just work because of the SDK's buoy-in environment variable support. Interested to hear how you get on!

[obscurerichard]

Unfortunately the experiments we did were a bust. We made sure that Minio had public Internet connectivity to all the right systems:

1. The workstation where gh-gei was run
2. the GHES instance that was the source, and
3. the full list of published CIDRs for github_enterprise_importer from https://api.github.com/meta

However, we could not get it to work. Perhaps we were missing some key configuration element, but this round of experiments did not confirm compatibility.

[timrogers]

Are you able to share some log output, perhaps with me privately on [email protected]? I'd like to see what happened.