pre-commit-terraform
pre-commit-terraform copied to clipboard
Add a feature to clone modules via SSH keys during terraform_validate stage
It would be great to add a feature to be able to clone modules via SSH keys during terraform_validate stage.
That's from https://ukrops.slack.com/archives/C3R001CHW/p1659634407658399
TL;DR Possible solution: apk add --no-cache openssh
in the last Dockefile stage.
@MaxymVlasov @yermulnik Wanted to get your thoughts on implementation here before I start working this...I think this moves out of just the terraform_validate
hook, since terraform init
gets called in common::terraform_init
by at least terraform_validate
and terraform_providers_lock
currently. Please let me know if I'm missing a simpler solution here.
Primary issues are getting SSH keys and public SSH host keys inside the container.
I think my preference initially is to provide ssh-agent
access via docker bind (and having a section in the README about running the container with SSH access), but not support binding the user's .ssh directory (since those keys would be more likely to have a passphrase). And for the known_hosts, I think the 2nd option I list below (ssh-keyscan
), though needing more code in _common.sh
, is better than binding a known_hosts
file as the hooks will then run even if the user has not previously connected to the host.
For keys:
- If
ssh-agent
is running when docker is called, we'd probably want to give access to that instead of trying to copy in key files (which may not even exist in the case of running the container on a remote system, wheressh-agent
access is passed throughssh -A
or similar). So having the user execute something like:
docker run -v ${SSH_AUTH_SOCK}:/run/ssh.socket -e SSH_AUTH_SOCK=/run/ssh.socket
would be necessary.
- If there's no
ssh-agent
, the user would have to provide a passphraseless key, either by binding it via thedocker run
or maybe passing via an environment option, but neither of these seem ideal.
For public host keys:
- Require the user to provide a known_hosts file that contains the remote host signatures via binding
or
- Before running
terraform init
, we have to scan for anysource =
URLs that match the three SSH styles supported by terraform:
source = "[email protected]:hashicorp/example.git"
source = "git::ssh://[email protected]/storage.git"
source = "git::[email protected]:storage.git"
it's easy enough to parse out the hostnames from these, and run ssh-keyscan
with output to a temporary known_hosts file. But it does add complexity to the common::terraform_init
function. This would work even if the user is using a private custom SSH git repo.
@gartemiev do you think this would meet your needs? Is there a case that makes more sense to you?
Well, yeah, that's a bit of a hassle π€
Mounting auth socket would mean some of the headless setups might fail as they might not have ssh agent running (or ssh keys added to it).
On the other hand mounting .ssh
dir (or explicitly priv/pub keys only) would mean a need to provide a passphrase if key requires it, which again might be a point of failure for headless setups.
There's another option to globally replace GIT URLs to use https://
, which might not work for some setups with self-hosted GIT implementations where access via HTTPS isn't provided.
I'm leaning to an option to mount .ssh
dir (or explicitly priv/pub keys only) along with requirement for users to provide passphrase-less key in case they're running pre-commit-terraform
Docker container against GitHub-hosted Terraform modules. This would allow headless setups to get provided with such a key to access GH via SSH protocol. Kinda a big caution message in the README on how to do that with as less hassle as possible (plus what Max wrote re ssh installation into Docker image).
What I actually mean is that we should not interfere with any user-related stuff, but only provide an option on how to overcome GitHub's limitation with user-side actions.
Does it make sense to you folks?
I spent some time experimenting with this, and I think @yermulnik's plan for just providing documentation on how to solve this (with openssh-client
installed in the image) is workable. Especially trying to follow @MaxymVlasov's thoughts on nocode. Depending on what exactly the user needs, it might take some setup on their side, but once setup I think it would work well. We could also provide a doc section on binding the ssh-agent
socket where useful.
We could put a simple plan in the README
, but having a full explanation on some of the setup options may necessitate a separate file (or a wiki entry if the project had one).
I've put notes below (primarily for myself), but if agreed that @yermulnik's plan makes sense, I can put together a pass at docs and the minor Dockerfile
change. What are thoughts on having a separate file for more thorough documentation vs a wiki?
Notes
For host keys, while not ideal I think using SSH option StrictHostKeyChecking=no
might be acceptable, since we're already using passphraseless keys.
If the user wants to bind their existing .ssh
directory, and not have to modify their config, they can use -e GIT_SSH_COMMAND="/usr/bin/ssh -o StrictHostKeyChecking=no"
when running the container, and the host key problem should be resolved.
On Github this plan would allow using read-only repository level deployment keys, or account level SSH keys. Other systems (private hosted git repos, Codecommit, etc) shouldn't have an issue either. One issue with Github repository deployment keys is they have to be unique per repository, so if there are multiple private github repos referenced in the code, the source =
statements have to refer to a separate Host
entries for each repository. I put an example of how I think this would have to be handled below.
As an example, let's say I have 2 Github repositories, tofupup/private-tf-security-group
and tofupup/private-tf-vpc
. For read-only deployment keys for each repository:
- (optional) create a
.ssh
directory for this project
$ PROJSSH="/home/john/src/proj-sshdir"
$ mkdir -p $PROJSSH
- Create passphraseless SSH keys for both repos:
$ ssh-keygen -t ed25519 -C "github_security_group_buildkey" -N "" -f "$PROJSSH/id_github_security_group_buildkey"
$ ssh-keygen -t ed25519 -C "github_vpc_buildkey" -N "" -f "$PROJSSH/id_github_vpc_buildkey"
- Add keys to respective repositories (via web or
gh
CLI)
$ gh repo deploy-key add "$PROJSSH/id_github_security_group_buildkey.pub" -t "john security_group buildkey" -R tofupup/private-tf-security-group
$ gh repo deploy-key add "$PROJSSH/id_github_vpc_buildkey.pub" -t "john vpc buildkey" -R tofupup/private-tf-vpc
- Create ssh config
β― cat $PROJSSH/config
Host gh_security_group
Hostname github.com
ControlMaster no
IdentitiesOnly yes
IdentityFile ~/.ssh/id_github_security_group_buildkey
StrictHostKeyChecking no
Host gh_vpc
Hostname github.com
ControlMaster no
IdentitiesOnly yes
IdentityFile ~/.ssh/id_github_vpc_buildkey
StrictHostKeyChecking no
-
source
statements in module must reference these new hostnames:
source = "git::ssh://git@gh_security_group/tofupup/private-tf-security-group"
source = "git::ssh://git@gh_vpc/tofupup/private-tf-vpc"
A separate file is OK, put it in the docs/ folder.
Wiki has lack of search, so for me it is useless.
I could introduce the mkdocs->gh-pages integration later, if we will have many docs (I'll already have needed skeleton and settings, so it should be done in less than 4 hours, when the time comes)
+1 to file instead of wiki.
@tofupup Thanks for your time and effort. What you wrote in the latest post looks reasonable and good to me.
The only usual nitpick bit would be to wrap var into double quotes mkdir -p "$PROJSSH"
π
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days