CDCgov
Importing JosiahSiegel GitHub Actions into project
The goal is to import these GitHub Actions (remote repos) developed in Mr. Siegel GitHub personal account into the CDC ReportStream project as local actions.
├── AzViz-action
├── action-connect-ovpn
├── checksum-validate-action
├── git-secrets
├── randomrepo
├── rapid-wsl
├── reliable-pull-request-action
├── remote-branch-action
├── runleaks
├── slack-boltjs-app
├── stackoverflow_in_pg
├── terraform-stats
├── terraform-templates
└── workflow-housekeeper
Based on @devopsmatt recommendation, I will focus on just fitting these GitHub Actions into the already established format on GH and elsewhere. Based on the maturity of the project we are assuming these GHA do not require any additional development.
Note: I am only concerned that these specific actions are directed to unique commits and not to the latest-tag, release or versions. This was the reason for my initial plans to contact @JosiahSiegel and inquire about it.
It seems that there is either a previous attempt to import these GitHub Actions or what already exists in the .github/actions folder is required to interface with the remote-action. I am not sure yet of what these are:
- .github/actions/vpn-azure
- .github/runleaks
- .github/vpn
I can either spend time trying to figure out what is going on (which is part of a larger initiative to learn about the framework) or simply effectively save time and directly ask Mr. Siegel about it.
Note: I will continue to review these GHA but time is limited and I have very little time to waste since the devops team is leaving. I will take the path of directly asking Mr. Siegel if he can provide me with some insights.
One thing seems to be clear at this point. This is not a simple matter of importing a remote GHA but rather a more elaborated structure where some local configurations are needed and they might need to be changed/modified to accommodate the importing of these remote GHA.
@emvaldes all github actions are kept updated by dependabot and use specific commits to prevent projects from switching underlying code under the guise of the same tag. This is a security requirement. @snesm
The original idea was to take the easy/quick approach of forking JosiahSiegel repos into the CDC Organization and then update the GitHub Actions use entries to point to the newly forked repos.
This is the best solution we could possibly have as we would be maintaining a functional path to update our repos with the upstream development efforts. This is a less intrusive/disruptive change to the existing workflows.
I was going to engage the CDC GitHub Organization and request for them to fork the repos from JosiahSiegel. Note: This request might take some time but meanwhile we can use those forked repos.
At some point, @devopsmatt clarified his vision and explained that this approach although more holistic will introduce a set of additional repositories to be managed inside the CDC Organization and we should not extend that complexity.
His vision is to simplify these efforts and make these features to be part of the project so that inheriting this project will be as self-contained as possible.
While this is resolved, I will go ahead and just include these original repos into the project's .github/actions section and leave them there as placeholders. Just to make sure they are part of the codebase and tied to the project.
I have now created a branch named "importing-gha" and injected the three repositories into the .github/actions folder as placeholders for completing the importing process.
devops: prime-reportstream (master=) $ git checkout -b importing-gha ; Switched to a new branch 'importing-gha'
devops: prime-reportstream (importing-gha) $ git status ;
On branch importing-gha
Untracked files:
(use "git add
nothing added to commit but untracked files present (use "git add" to track)`
devops: prime-reportstream (importing-gha) $ git status ; On branch importing-gha nothing to commit, working tree clean
devops: prime-reportstream (importing-gha) $ git push --set-upstream origin importing-gha ;
Username for 'https://github.com/CDCgov/prime-reportstream.git': emvaldes
Password for 'https://[email protected]/CDCgov/prime-reportstream.git':
Enumerating objects: 58, done.
Counting objects: 100% (58/58), done.
Delta compression using up to 16 threads
Compressing objects: 100% (49/49), done.
Writing objects: 100% (55/55), 19.21 KiB | 3.84 MiB/s, done.
Total 55 (delta 11), reused 1 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (11/11), completed with 3 local objects.
remote:
remote: Create a pull request for 'importing-gha' on GitHub by visiting:
remote: https://github.com/CDCgov/prime-reportstream/pull/new/importing-gha
remote:
remote: GitHub found 7 vulnerabilities on CDCgov/prime-reportstream's default branch (2 high, 5 moderate). To find out more, visit:
remote: https://github.com/CDCgov/prime-reportstream/security/dependabot
remote:
To https://github.com/CDCgov/prime-reportstream.git
- [new branch] importing-gha -> importing-gha branch 'importing-gha' set up to track 'origin/importing-gha'.
After importing all Josiah Siegel's GHA into the project, I have now modified the only two references within the GitHub Actions file structure pointing to these repos.
GitHub Actions:
./.github/actions/deploy-backend/action.yml:336: uses: JosiahSiegel/checksum-validate-action@ebdf8c12c00912d18de93c483b935d51582f9236 -- ./.github/actions/vpn-azure/action.yml:43: - uses: josiahsiegel/action-connect-ovpn@794339aff94452216c97f609476c367a43a31295
GitHub Workflows:
./.github/workflows/alert_terraform_changes.yml:32: uses: josiahsiegel/terraform-stats@68b8cbe42c494333fbf6f8d90ac86da1fb69dcc2 -- ./.github/workflows/deploy_terraform.yml:55: uses: josiahsiegel/terraform-stats@68b8cbe42c494333fbf6f8d90ac86da1fb69dcc2 -- ./.github/workflows/log_management.yml:16: uses: JosiahSiegel/workflow-housekeeper@731cc20bb613208b34efb6ac74aab4ba147abb50 -- ./.github/workflows/log_management.yml:28: uses: JosiahSiegel/workflow-housekeeper@731cc20bb613208b34efb6ac74aab4ba147abb50 -- ./.github/workflows/prepare_deployment_branch.yaml:31: uses: JosiahSiegel/remote-branch-action@dbe7a2138eb064fbfdb980abee918091a7501fbe -- ./.github/workflows/prepare_deployment_branch.yaml:40: uses: JosiahSiegel/reliable-pull-request-action@ae8d0c88126329ee363a35392793d0bc94cb82e7 -- ./.github/workflows/release_to_azure.yml:148: uses: JosiahSiegel/checksum-validate-action@ebdf8c12c00912d18de93c483b935d51582f9236 -- ./.github/workflows/scan_action_logs.yml:16: uses: josiahsiegel/runleaks@4dd30d107c03b6ade87978e10c94a77015e488f9
Note: I have left markers in these files to identify which current actions and workflows make use of these remote repos. We need to modify them at some point when our testing will yield functional results.
@devopsmatt has expressed his vision on not altering any of these actions and workflows but rather create new ones to emulate the changes we intend to have done. I agree on this approach and will continue to implement the basis for these efforts.
Once that is completed, then we can either replace the exiting workflows and actions calling or just create new ones. The more we will explore these configurations we will have a better understanding on what needs to be done as long as it's not disrupting the exiting processes.
There are now changes I have to perform to update self and cross referencing issues within the repos themselves (they are minimum changes to be done). After that, the next step is to identify existing and/or missing references to INPUTS, SECRETS and ENV variables.
Once that is resolved, then we can switch on these replacements. Until then, nothing will work and cannot be enabled to avoid breaking the build process.
We have to import 14 externally repositories (residing on Mr. Josiah Siegel's personal repos) and we need to identify them (temporarily) with some sort of prefix until we can normalize their naming patterns within the prime-reportstream GitHub Actions conventions.
For that, I will rename these repos so that it's clear the purpose of those within the GHA folder. e.g.: josiahsiegel--***
Note: This is for now while I work on these within our implementation branch (importing-gha). Before merging into MASTER we should have a decision made in this regard.
We have now identified and documented all secrets and variables used in these GitHub Actions. We are currently reviewing which of these already exists in this project and which ones are missing.
The goal here is to make sure that all parameters defined in the imported GitHub Actions are present in the Prime-ReportStream project. Once all missing items are identified, we will proceed to contact the repos maintainer (Mr. Joisah Siegel) and ask for his guidance to the configure the missing components so the import process basic onboarding is completed.
We have a lot to work on after these steps are completed because the import itself is not enough and we need to create new GitHub Workflows to tests these imported GitHub Actions.
It's now decided to disregard all the Josiah Siegel's external/remote repos that we initially targeted to import as it seems (based on his input) that only the "action-connect-ovpn" GitHub Action is worth importing.
devops: actions (importing-gha $=) $ git rm -r --cached ./azviz-action ./checksum-validate-action ./git-secrets ./randomrepo ./rapid-wsl ./reliable-pull-request-action ./remote-branch-action ./runleaks ./slack-boltjs-app ./stackoverflow_in_pg ./terraform-stats ./terraform-templates ./workflow-housekeeper ;
rm '.github/actions/azviz-action/.github/workflows/sample_full_workflow.yml'
...
rm '.github/actions/checksum-validate-action/.github/dependabot.yml'
...
rm '.github/actions/git-secrets/.gitattributes'
...
rm '.github/actions/randomrepo/.github/workflows/main.yml'
...
rm '.github/actions/rapid-wsl/.gitignore'
...
rm '.github/actions/reliable-pull-request-action/.github/workflows/test-action.yml'
...
rm '.github/actions/remote-branch-action/.github/workflows/test-action.yml'
...
rm '.github/actions/runleaks/.github/dependabot.yml'
...
rm '.github/actions/slack-boltjs-app/.dockerignore'
...
rm '.github/actions/stackoverflow_in_pg/.gitignore'
...
rm '.github/actions/terraform-stats/.github/ISSUE_TEMPLATE/bug_report.md'
...
rm '.github/actions/terraform-templates/.devcontainer/Dockerfile'
...
rm '.github/actions/terraform-templates/azure/modules/azure_ad/v2/files/winrm.ps1'
...
rm '.github/actions/workflow-housekeeper/.github/dependabot.yml'
...
Based on Josiah Siegel's recommendation:
[...] I was just saying the only critical action is the vpn one. All others support some other optional/nice-to-have feature (and can be replaced if you want to).
You can treat my actions like you treat the hundreds of gradle public libraries used by reportstream or copy and paste their guts directly into the project if you no longer wish to use the public source.
For public libraries, there are variables used in test build workflows that users do not need to be aware of. It is the same thing here. The only thing that matters is the input/output of the action.yml and reportstream is already doing that correctly. If you import the action repos, you still only need to use the action.tml input/output as they currently are within the reportstream repo. [...]
devops: actions (importing-gha +$=) $ git add . && git commit -m "Restructuring the scope for target GitHub Actions to be imported" ;
[importing-gha ae9f58a04] Restructuring the scope for target GitHub Actions to be imported
289 files changed, 17833 deletions(-)
delete mode 100644 .github/actions/azviz-action/.github/workflows/sample_full_workflow.yml
...
delete mode 100644 .github/actions/git-secrets/.gitattributes
...
delete mode 100644 .github/actions/randomrepo/.github/workflows/main.yml
...
delete mode 100644 .github/actions/rapid-wsl/.gitignore
...
delete mode 100644 .github/actions/reliable-pull-request-action/.github/workflows/test-action.yml
...
delete mode 100644 .github/actions/remote-branch-action/.github/workflows/test-action.yml
...
delete mode 100644 .github/actions/runleaks/.github/dependabot.yml
...
delete mode 100644 .github/actions/slack-boltjs-app/.dockerignore
...
delete mode 100755 .github/actions/stackoverflow_in_pg/.gitignore
...
delete mode 100644 .github/actions/terraform-stats/.github/ISSUE_TEMPLATE/bug_report.md
...
delete mode 100644 .github/actions/terraform-templates/.devcontainer/Dockerfile
...
delete mode 100644 .github/actions/workflow-housekeeper/.github/dependabot.yml
...