gh-gei
gh-gei copied to clipboard
FINAL: Git storage option feature
Summary:
This PR introduces the ability for the GEI CLI to upload archives to GitHub-owned storage using multipart uploads. To enable this, a new CLI option, --use-github-storage, has been added. This option allows users to explicitly specify that their archives should be uploaded to GitHub's managed storage instead of their own.
Key Features:
Multipart Upload Support: Leveraging multipart upload for reliable and efficient large file uploads. This ensures the robustness of the uploads, minimizing the chances of failure during transmission. We're investigating libraries to handle multipart uploads, but for now, this feature is not rolled by hand unless necessary.
New Option: --use-github-storage:
This option is required to upload archives to GitHub-owned storage and will be hidden until GitHub-owned storage reaches general availability (GA). By default, archives are still uploaded to user-specified storage locations unless the --use-github-storage flag is explicitly set.
Motivation:
As discussed during our EDR, we want to allow users to utilize GitHub-owned storage as an option without making it the default. This approach lets us roll out GitHub-managed storage in stages and gather feedback while providing an explicit way to opt-in to this feature.
Check in BBS MigrateRepoCommandArgs removed, relating to changes in this PR: https://github.com/github/gh-gei/pull/1057
Take it out for a 🚗 :
*** Remember to export env variables***
export GH_SOURCE_PAT=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
export GH_PAT=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
Migrating from GitHub Enterprise Server:
dotnet run --project src/gei/gei.csproj -- migrate-repo --github-source-org valet-testing --source-repo integration-tests --github-target-org octoshift-staging --target-repo YOU_ADD_NEW_NAME --use-github-storage true --ghes-api-url https://octoshift-ghe.westus2.cloudapp.azure.com/api/v3 --verbose
Successful migration_id: RM_kgHaACRlZjNiMmExMy02ZTRkLTQ0NWYtOTc4Yy05YTk1MTQ5NzE2OTk
Migrating from GitHub Enterprise Server- Multi part upload:
dotnet run --project src/gei/gei.csproj -- migrate-repo --github-source-org github-owned-storage --source-repo large-repo-10gb --github-target-org octoshift-staging --target-repo YOU_ADD_NEW_NAME --use-github-storage true --ghes-api-url https://octoshift-ghe.westus2.cloudapp.azure.com/api/v3 --verbose
```dotnet run --project src/gei/gei.csproj -- migrate-repo --github-source-org import-testing --source-repo large-repo --github-target-org octoshift-staging --target-repo 10-gb-multipart-updated --use-github-storage true --ghes-api-url https://octoshift-ghe.westus2.cloudapp.azure.com/api/v3 --verbose
Successful migration_id: RM_kgHaACRjZDJlMjEyNy03MDIyLTRlOWItYjY3Mi1jM2MzYmMwZjY3OTg
-------------------------------------------------------------------------------------------------------------
**Migrating from BBS:**
Source repo: https://test-bbs-o.githubapp.com/projects/IM/repos/codeowners-test/browse
You'll need to add yourself to the pinhole firewall at https://portal.azure.com/?wa=wsignin1.0#@githubazure.onmicrosoft.com/resource/subscriptions/7c45ec63-636b-445b-8185-54accbc0190d/resourceGroups/octoshift-test-bbs/providers/Microsoft.Compute/virtualMachines/bbs-test/networkSettings
Here's the SSH key (and other creds) for our bbs-test-o instance: https://start.1password.com/open/i?a=LKXPAKKGYNBF7IOHIU6VPDFEU4&v=akrfryl3erhcqdcvgiqqtvh254&i=23dceluq4vdvpptr7qen3wcbiu&h=github.1password.com
dotnet run --project src/bbs2gh/bbs2gh.csproj -- migrate-repo --bbs-project IM --bbs-repo codeowners-test --ssh-user bbs --ssh-private-key /Users/begonaguereca/.ssh/id_rsa_bbs_octoshift --use-github-storage true --bbs-server-url https://test-bbs-o.githubapp.com --github-org octoshift-staging --github-repo ADD_NEW_NAME --verbose
Successful migration_id: RM_kgHaACRhODI4NWRmZS0yNWM3LTQ1YjEtYTJhOC1jYmZjNTkxZDE0Mjk
**Migrating from BBS:**
Successful migration_id:
-------------------------------------------------------------------------------------------------------------
Closes: https://github.ghe.com/github/octoshift/issues/8969
<!--
For the checkboxes below you must check each one to indicate that you either did the relevant task, or considered it and decided there was nothing that needed doing
-->
- [ ] Did you write/update appropriate tests
- [ ] Release notes updated (if appropriate)
- [ ] Appropriate logging output
- [ ] Issue linked
- [ ] Docs updated (or issue created)
- [ ] New package licenses are added to `ThirdPartyNotices.txt` (if applicable)
<!--
For docs we should review the docs at:
https://docs.github.com/en/migrations/using-github-enterprise-importer
and the README.md in this repo
If a doc update is required based on the changes in this PR, it is sufficient to create an issue and link to it here. The doc update can be made later/separately.
-->