[AINFRA-1533] Adopt `git-conceal` in this repo
Closes: AINFRA-1533
What
This PR migrates the repo to use git-conceal instead of configure_apply, so that the usage of secrets in the repo gets easier and more transparent for developers.
How
- 🗑️ Delete
.configureand.configure-files/*.encfiles - Add files that were listed in
.configureas encrypted files in the repo instead, by also adding their path as entries in the.gitattributeswithfilter=git-conceal diff=git-conceal—so they get encrypted bygit-concealon push - Update
.gradlefiles andFastfileto account for new file paths - Update
.gradlecode to account for the fact that now the files are present (but encrypted) after cloning the repo—as opposed to being absent when we didn't runconfigure_applyin the old way. - Replace calls to
bundle exec fastlane run configure_applyin.buildkite/*commands—and calls toconfigure_apply(force: true)inFastfile—with calls togit-conceal-unlock, and bumpci-toolkitplugin to point to https://github.com/Automattic/a8c-ci-toolkit-buildkite-plugin/pull/195 where this helper lives - 📚 Updated documentation
[!NOTE]
ℹ️ About the git-conceal-helper helper from ci-toolkit
The way this currently works is that:
- The
git-concealGitHub repo builds new binaries for macOS, Linux and Windows when we do a new release, and attach them as assets to the GitHub Release- The
git-concealGitHub repo also provides aninstall.shscript to make it easy to installgit-concealfor your current platform (the script takes care of detecting if you're on a Linux, macOS or Windows agent, downloads the right asset from the latest GitHub release for that platform,chmod +xit and install it)- The
git-conceal-helperwrapper from oura8c-ci-toolkitplugin is then just a small bootstrap/helper script that checks ifgit-concealis installed in the CI agent, if so callsgit-conceal unlock "env:${1:-GIT_CONCEAL_SECRET_KEY}"and exit; otherwise calls theinstall.shscript ofgit-concealto install it first before callinggit-conceal unlockThe way this might work in the future:
- We will probably ship
git-concealin most of our self-hosted CI agents and custom AMIs to avoid having to download it on each job… even if in practice it takes less than 1s to download and unlock the repo- We will still keep that wrapper script in
ci-toolkitthat does the "tests if installed already and install it if not" logic, because not all our CI agents are custom agents (e.g. we sometimes use thedefaultqueue which uses EC2 instances with the official Buildkite AMI, not customized by us), so that would still be useful for those.
Merge timing
[!IMPORTANT] Do not merge this PR until (1) we have all the documentation / FAQ ready for all devs to follow in FG and (2) we have validated similar PRs in other repos—especially ones using macOS/Windows and Tumblr CI agents—work too..
Test Steps
General check
- Verify CI is green
- Checkout this branch and validate that all the secret files added to the repo in this PR are encrypted and unreadable:
WooCommerce/google-services.jsonWooCommerce/upload.jksdebug.keystorefirebase.secrets.jsongoogle-upload-credentials.jsonsecrets.propertiessentry.properties
- Review the documentation updates
- Review and ensure all the changes necessary in
*.gradlefiles to reflect the new setup are covered by this PR (path updates to.propertiesor keystore files, code updates around logic that was based on if a secret file exists or not, …)
Follow the README.md instructions as an Automattician
- Clone the repo from scratch in a new directory on your Mac, then checkout this PR's branch
- Copy the git-conceal encryption key from our Secret Store in your clipboard, then run
pbpaste | base64 -d | git-conceal unlock -at the root of your new clone's directory - Validate that after this step, you are able to read the content of secret files like
secrets.propertiesin clear now - Build and Run the project, validating everything works as expected (including e.g. Google login to validate
google-services.jsonis taken into account, etc)
[!NOTE]
ℹ️ The real migration won't require devs to clone the repo from scratch; this is just for the testing steps of this PR
Above I suggest to test this scenario in a fresh clone of the repo rather than your everyday working copy, especially to avoid risking to leave your git repo in a state that is setup for
git-concealwhile you were just testing and the PR is not merged yet.In practice, once the PR is merged in
trunk, devs won't need to clone the repo fresh to migrate togit-concealthough. They will just have to runpbpaste | base64 -d | git-conceal unlock -on their existing working copy once, and probably never run agit-concealcommand again afterwards (except maybegit-conceal statusif they want to validate a new secret file they're about to add is properly gonna be encrypted as they expect).At some point I'd still highly recommend for them also get rid of the old files that were installed by their previous setup with
configure(rm -rf ~/.configure/woocommerce-androidandgit clean -dxi -e .DS_Store -e .idea -e local.properties), to avoid confusion and risking relying on obsolete files.
Follow the README.md instructions as an external contributor
- Clone the repo from scratch in a new directory on your Mac, then checkout this PR's branch
- Validate files like
secrets.propertiesshow as encrypted garbage - Edit
defaults.propertiesto add the relevant values forwp.oauth.* - Overwrite
WooCommerce/google-services.jsonencrypted file withWooCommerce/google-services.json-example - Build and Run the project, validating everything works as expected (except Google login)