Support ssh git dependencies in pip requirements.txt when building with use-container

[johnc44]

Describe your idea/feature/enhancement

In a requirements.txt file I can have something like this:

git+ssh://[email protected]/path/to/my.git

If I do a sam build on this, it works. But if I add --use-container it does not.

Host key verification failed. fatal: Could not read from remote repository.

I presume this is a combination of my SSH key plus known_hosts not being taken from the host.

Proposal

The only 2 solutions I can think of are either to copy the keys onto the container (hmm) or to clone them on the host, copy them onto the container then run pip install on them.

Neither seems ideal to me (and I can see why you wouldn't want to do the first one), but I'm hoping that there is maybe something I've not thought of.

I'd like to continue to use --use-container but we also need a way to share code between lambdas and this seems to me to be the easiest. I don't know much about hosting a private pypi yet but I presume I'll run into a similar problem there.

[johnc44]

After a bit more rummaging around I found this: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

which sounds like it's at least theoretically possible to proxy ssh through the host.

[nickswiss]

Any update here? Seems like this should be supported? I am installing python packages which may build differently based on the operating system, so I use --use-container command to avoid this issue.

sam build --use-container

I have private python packages (hosted on enterprise github) which I need install via ssh. I feel like secrets are a very common necessity in docker builds so I'm either surprised or cannot find the documentation explaining how to do this.

The article posted by @johnc44 requires secrets so really if you guys added something which allowed passing secrets, or any docker flags alongsied the --use-container command, I'd greatly appreciate it.

[cmurtaugh]

I can build an image outside of SAM using a command like:

docker build --ssh build_ssh_key='/path/to/my/key' .

and then referencing that key in my Dockerfile like:

RUN --mount=type=ssh,id=build_ssh_key pip install -r requirements.txt 

But if I try to build the same app using SAM, adding this to my function resource:

    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./my_function
      DockerBuildArgs:
        ssh: "build_ssh_key='/path/to/my/key'"

the build fails with a publickey authentication error from GitHub.

[Macok]

I implemented this in #3084. With this change, the host's SSH configuration can be forwarded to the container by mounting ssh-agent socket.

sam build -u --container-dir-mount $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent --container-dir-mount ~/.ssh/known_hosts:/etc/ssh/ssh_known_hosts

Let's see if it gets approved.

[vacuumn]

Any updates on this?

[Jonathan-Landeed]

It seems the PR was closed in favor of using a docker image. I'm not sure exactly how a docker image is supposed to work without being a security issue, but here's a dockerfile that bakes in some ssh credentials:

FROM public.ecr.aws/sam/build-python3.10:latest-x86_64

RUN ssh-keyscan github.com > /etc/ssh/ssh_known_hosts

COPY deploy_key /root/.ssh/deploy_key

RUN chmod 600 /root/.ssh/deploy_key && \
    chmod 700 /root/.ssh

RUN echo "    IdentityFile /root/.ssh/deploy_key" >> /etc/ssh/ssh_config