Add SSH agent support via "keys: agent"

[sk33lz]

What is your lando version and operating system?

• Lando v3.0.0-beta.22 on MacOS Sierra 10.12.6

Feature/Enhancement Request

I use a password protected SSH key, and I'm sure many others who use lando, or would like to do. I did some research and apparently having to enter a password on every remote command is not something that is lando's fault per se. It's apparently a known issue with Docker and MacOS due to the way the MacOS handles sockets. Luckily, there seems to be some work being done on workarounds for this issue by other teams. Below is one that I found that looks like it could be integrated into the lando backend to run docker commands with extra parameters to allow for a single entry of the password while running commands like lando drush sql-sync on a remote server using drush aliases.

https://github.com/uber-common/docker-ssh-agent-forward

Other implementations I found in my searches:

• https://github.com/AaronFeledy/lando-scripts
• https://github.com/nardeas/docker-ssh-agent

I have currently setup a workaround on the lando project level by adding a custom tooling command to our .lando.yaml file. This command forwards the agent before each remote drush command, but it would be great to see something like this implemented in the lando commands on backend so that it's easier to use password protected SSH keys out of the box.

Thanks for a great tool!

• Jason

[pirog]

@sk33lz so im definitely open to an implementation (read: PR) for this. that said while lando does allow users to inject passphrase-protected keys our "official" (but flexible and not-dogmatic) position is that users should use "non passphrase protected keys" with lando. There are a few reasons for this

1. It is likely there will be future GUI products built on Lando and a design goal there is to only ask the user for their password once (and that is during the actual install and its their system password)
2. I don't personally think that a passphrase protected key that you intend to only enter once is significantly more secure for the local dev case than just having your computer itself have a password.

I'm going to backlog this and again, im super open to a PR that would support this but i dont think its something we can prioritize internally (at least right now)

[dnotes]

This is not a particularly important issue for me, but I'm wondering why you don't think passphrase protected keys with ssh-agent is significantly more secure than just having the computer have a password? I mean, if I lose my computer and someone gets into it, I would very much appreciate some additional barrier between them and full access to all of my git repositories and servers. In an age when exploits for obtaining computer access can be as easy as using "root" with no password and clicking the "login" button twice (nice one Apple), I think protecting one's SSH keys is absolutely obligatory for any developer.

Anyhow this would be nice eventually but it's not a huge issue for me to type in keys on rebuild.

[pirog]

I don't think it's significantly more secure if you intend to only enter the passphrase once as described in this issue eg in a keychain or some other password management solution that someone who has broken into your computer likely has access to. In this scenario you more or less have a passphrase-less key anyway.

If you do intend to enter the passphrase every single time you use the key then yah obviously that is more secure. That said, im guessing the vast majority of people use the same password for their laptop as they do their passphrase protected key, so the most important thing you can do to protect yourself is having a good password for your machine.

It's also worth pointing out that you still definitely can use passphrase protected keys with lando, its just not the default functionality.

[sk33lz]

I work locally with Lando, so the idea was that entering it once was more of a usability enhancement rather than a security concern. Entering the password once was just an easier workflow to deal with for local development.

The tooling workaround I am using is working ok for now, but it would be great to see something out of the box down the line to support one of the fixes mentioned in my original feature request. I'll submit a PR if I come up with a good solution to contribute. I'd love to leave this open to see if anyone else in the Lando community comes up with a workable solution for this.

Thanks again!

[pirog]

@sk33lz yup. totally understand that. my point there was simply it seems like a lot of extra effort on the lando end to implement with not a ton of extra benefit (eg going from entering no password to entering it just once).

That said im not opposed to making a change that would allow this its just something Team Lando can't focus on in the foreseeable future, aka its likely this would have to be a community provided feature in the short-medium term.

[jaydansand]

SSH agent support is far more than just an extra one-time password - that's a dangerous oversimplification, so I'd like to pose a few points against that security assumption. Also, because I really like Lando and want SSH agent forwarding to work easily.

If you store unencrypted private keys on your machine, then they are vulnerable to exfiltration by virus/accident (and subsequent abuse), or via OS login bypass (as @sk33lz mentioned), and a host of other issues. Once the private key is out in the wild, it can be used against you until you go through all of your servers and revoke the key.

With an SSH agent (and encrypted keys), the keys are never directly presented to any process on the computer - the agent negotiates connections, and if the agent is well-written, the decrypted keys are stored in private, non-paged memory so it won't reside anywhere (swap file etc.) once the computer is turned off. That means exfiltrated key files are inert because they're still encrypted. Other processes can't harvest my keys because the agent doesn't leak them. If a virus is running on my computer (and it didn't keylog, or at least didn't correlate my key password), it can only abuse my keys for SSH connections while the key agent is running.

Another benefit of SSH agents is forwarding. I have servers from which I need to use git or rsync over SSH. I want to disable passphrase-based login on those boxes, but that means I'd have to put my private key on a bunch of servers so that I can leapfrog from one host to another. Enter SSH agents! I keep encrypted keys on my dev box, load them into my agent at the start of the work day, and forward my agent across SSH connections so that my keys are as safe as possible and SSH/git/rsync work transparently and without any password prompts from one server to another. No server ever needs to have a copy of my private key on disk.

@sk33lz I'd love to hear how you got Lando to forward your agent - I'm using git within Lando (because RHEL 7 still ships with git 1.x) and it's a pain to push/pull without ready access to my keys. Philosophically, I don't want a dev box that other people can root/sudo on to have a copy of my private key.

[ba66e77]

How is this different from what was requested and merged in #344 ?

[lslinnet]

@ba66e77 The difference is that by passing a key into the container like #344 is doing you will have to enter the password everytime your composer install, npm install or any other tool on the box needs to make a connection to one of the packages that is private or require a certain key to obtain.

Additionally as @jaydansand pointed out the forwarding of ssh-agent allows you to connect to one server and then from that server use your ssh key to connect to the next in turn. Common place this is used is a Jump-host (example being connecting to Jumphost to continue on to the non-public mysql/elasticsearch/memcache/solr server)

@sk33lz I would love some insights into how you have configured your lando setup to allow for this ssh-agent forwarding you mention (I am guessing using the pinata ssh forward approach, but can't quite figure out how).

[sk33lz]

@ba66e77 It looks like that might resolve this issue. I'll have to try it and I'll follow up here.

@jaydansand @lslinnet I basically added a script that gets called as part of a Lando tooling command that adds the agent prior to running the command and you only have to enter it once. It's not great, but it's better than having to enter it like 5 times when trying to run a single lando command. Here is one I have for sql-sync that is run with the command lando prod-sql-sync.

Add this code to your .lando.yml file.

tooling:
  prod-sql-sync:
    description: Sync db from prod to local with fewer ssh password prompts.
    service: appserver
    cmd: /app/scripts/prod-sql-sync.sh

Add this code to a scripts dir off the root of your repository. You could have this ignored in your .gitignore if you want.

#! /usr/bin/env bash

SSH_KEY_FILE='/user/.ssh/id_rsa'
eval `ssh-agent` > /dev/null
ssh-add $SSH_KEY_FILE

drush sql-sync @site.prod @site.local

[pirog]

Alright wanted to kickstart the discussion over here again to gauge interest.

Now that we are through some of the larger foundational work to get Lando ready for the official 3.0.0 we can start looking for a few additional features to try and get rolling before that glorious day of deliverance. I personally think some level of ssh-agent support would be a good idea here.

Does anyone here eg @sk33lz or @dnotes or @ba66e77 or @jaydansand or @dustinleblanc or @serundeputy have a proposal on how to do that that would require the least amount of changes and moving parts on the lando side of things?

Also note that as of Lando 3.0.0-rc.2 we are injecting ALL keys into each container, including passphrase protected ones.

[lslinnet]

@pirog I haven't tried out 3.0.0-rc.2 yet, but do you by injecting also mean that a ssh-agent gets started in the containers?

[pirog]

@lslinnet no, i just mean that passphrase protected keys are available in the containers now. The comment at https://github.com/lando/lando/issues/478#issuecomment-463770730 was basically an RFP for how to take all those injected keys and make sure they are loaded into an ssh-agent that is accessible in each apps containers

[geerlingguy]

Just asking, since I was kind of roped into an outside discussion about this issue: how are the keys made available inside the containers? One of the concerns raised earlier in the thread was about one of the security protection SSH Agent offers is the fact that it is much harder (or impossible) to exfiltrate SSH keys off a local workstation if password protection is used.

But even if not, I can use SSH Agent and my keys never hit disk / readable volumes on a remote server (or in this case docker container), so no applications running inside that server/container could conceivably access them unless they also have access to the SSH Agent itself.

Is the current implementation (as of rc2) copying SSH keys from ~/.ssh into the container? Or is it looking at SSH Agent, reading out the list of loaded keys, and copying those in? Or is it a volume mount?

If so that seems like a major security risk, as the Docker images used for various services are not all owned by lando on GitHub, so I would need to be monitoring all the upstream images for potential security vulns that could allow the exfiltration of my SSH keys through one of the containers. (E.g. MySQL uses bitnami/mysql by default, Apache uses bitnami/apache—it looks like PHP at least uses a Lando-supplied image devwithlando/php).

[dnotes]

I agree with Jeff; copying ssh keys into containers worries me.

For another approach, check out how Docksal does this; they have an ssh-agent service container which shares ssh-agent proxy connections with all other containers.

It may also be possible to just share the ssh-agent sockets directly with individual docker containers—one would be asked for passwords with each project restart, which seems ok or better—and you can find documentation for this approach if you search for ssh-agent docker container or something like that. Basically it's just a --volume share, but there are some intricacies that I don't know about.

I'm not a devops or security expert, and I don't know the ramifications of this, but intuitively I feel that either one of these options would be better than copying ssh keys into containers.

On Tue, Feb 19, 2019 at 10:59 AM Jeff Geerling [email protected] wrote:

Just asking, since I was kind of roped into a discussion about this issue: how are the keys made available inside the containers? One of the concerns raised earlier in the thread was about one of the security protection SSH Agent offers is the fact that it is much harder (or impossible) to exfiltrate SSH keys off a local workstation if password protection is used.

But even if not, I can use SSH Agent and my keys never hit disk / readable volumes on a remote server (or in this case docker container), so no applications running inside that server/container could conceivably access them unless they also have access to the SSH Agent itself.

Is the current implementation (as of rc2) copying SSH keys from ~/.ssh into the container? Or is it looking at SSH Agent, reading out the list of loaded keys, and copying those in? Or is it a volume mount?

If so that seems like a major security risk, as the Docker images used for various services are not all owned by lando on GitHub, so I would need to be monitoring all the upstream images for potential security vulns that could allow the exfiltration of my SSH keys through one of the containers. (E.g. MySQL uses bitnami/mysql by default, Apache uses bitnami/apache—it looks like PHP at least uses a Lando-supplied image devwithlando/php).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lando/lando/issues/478#issuecomment-465265177, or mute the thread https://github.com/notifications/unsubscribe-auth/AAdyEioOKSVqR0PlH8q7sUjvJsWQffoKks5vPEl0gaJpZM4QW6nn .

[thursdaybw]

In sk33lz's opening comment he said:

I have currently setup a workaround on the lando project level by adding a custom tooling command to our .lando.yaml file. This command forwards the agent before each remote drush command It took me a lot of googling and trial and error to work out what that actually meant.

In my case I call a script from a tooling command that clones another repo and then pulls in gitsubmodules and I was being asked for the passphrase on every submodule.. here is my workaround:

The tooling command

 'moodle-install':
    description: Clones moodle into the correct directory and writes config.php with correct parameters
    cmd:
       - appserver: bash /app/.lando/scripts/moodle-install.sh

SSH Agent setup at start of moodle-install.sh

eval `ssh-agent -s` # start the ssh agent
ssh-add /user/.ssh/id_rsa # add the key to the ssh-agent, we are asked for the passphrase at this point, and then not again for this session.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[dnotes]

FYI, this is the main reason why I switched to Docksal. I actually like Lando better but this is a deal-breaker for me.

On Sun, Apr 21, 2019 at 11:42 AM stale[bot] [email protected] wrote:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea if you are wondering why we auto close issues.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lando/lando/issues/478#issuecomment-485273376, or mute the thread https://github.com/notifications/unsubscribe-auth/AADXEET4WT72U7BYCMX7GVDPRSYPZANCNFSM4EC3VHTQ .

[pirog]

Then it sounds like you’ve made a principled choice! We hope you enjoy docksal!

On Thu, Apr 25, 2019 at 7:56 PM David Hunt [email protected] wrote:

FYI, this is the main reason why I switched to Docksal. I actually like Lando better but this is a deal-breaker for me.

On Sun, Apr 21, 2019 at 11:42 AM stale[bot] [email protected] wrote:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this < https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea

if you are wondering why we auto close issues.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lando/lando/issues/478#issuecomment-485273376, or mute the thread < https://github.com/notifications/unsubscribe-auth/AADXEET4WT72U7BYCMX7GVDPRSYPZANCNFSM4EC3VHTQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lando/lando/issues/478#issuecomment-486878145, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFOFUCUF6643YKB2JTENF3PSJAKRANCNFSM4EC3VHTQ .

[jaydansand]

I'm still very interested in this. It looks like some people had success with pure Docker via mounting the SSH agent socket into the container and setting appropriate environment variable in it: https://gist.github.com/d11wtq/8699521

I have no idea how hard that would be to achieve in Lando, or whether it should be an opt-in or just always assumed it should be forwarded if $SSH_AUTH_SOCK is defined. I really don't like the idea of copying keys from ~/.ssh and all of the security implications that has.

[calebdelnay]

I tested the following with the Drupal 8 recipe and it allowed me to use Composer (via lando composer) against some private git repositories by running ssh-agent on my host machine (Ubuntu 18.04) to which I've added my password protected private keys.

Note that if $SSH_AUTH_SOCK isn't set due to no ssh-agent running, building the config will not work, because $SSH_AUTH_SOCK will be treated as blank, and the volume mount syntax won't be valid. For me, this is a desired behavior, because I want to ensure I start ssh-agent before any Lando services.

I haven't tested this with functionality like drush sql-sync but it should work for any command that uses ssh under the hood.

services:
  appserver:
    overrides:
      volumes:
        - "$SSH_AUTH_SOCK:/ssh_auth_sock"
      environment:
        SSH_AUTH_SOCK: /ssh_auth_sock

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[pirog]

unstale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[pirog]

unstale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[pirog]

unstale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.