Proposal for a better sam local support of docker-machine

[lestephane]

Background

I use docker-machine as my docker daemon, so I have one VM per project, which I can destroy when I'm done, keeping my host system pristine and without docker installed.

I've been meaning to use SAM CLI, and so far I couldn't.

So I set out investigating, and here is the explanation of the problem, and a proposal for a solution that, if implemented, will save a lot of support issues and confusion amongs docker-machine users in future.

Maybe docker-machine afficionados know all this already, but it took me enough hours to figure out that I wanted to provide the canonical solution, to leave things in a better state than I found it.

And if you're not going to implement the proposed solution, then maybe you'll at least add it as a known problem with the suggested solution, as part of the documentation.

Problem

sam local expects the filesystem of the sam application source code to be visible to the docker daemon, which it is not when using docker-machine. docker-machine runs in a separate (usually VirtualBox) VM.

Taking the example of java8 HelloWorld:

2019-07-06 11:13:13 Mounting /tmp/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container

there are three environments at work

• my work machine, which contains the /tmp/sam-app source directory
• the docker-machine VM, which does not contain any /tmp/sam-app directory
• the container which has /var/task mount that points to the non-existent /tmp/sam-app directory in the docker-machine VM. ie /var/task is empty

The missing link is between my work machine /tmp/sam-app and the docker-machine VM /tmp/sam-app. Unless the user takes extra steps to share/mount/sync/ the source code directory under the same path and with read permissions to the docker daemon in the docker-machine VM, the /var/task mount will be empty.

A couple issues I've seen related to this problem are:

• #1126: The problem disappeared when the reporter stopped using docker-toolbox, which is basically docker-machine
• #571: The one that I first encountered, ClassNotFoundException there basically means the sam source directory is nowhere to be found in the docker-machine VM, it is not a permission issue, as I was led to believe.

Since there is no obvious way from the output of sam local to determine whether a user is using docker-machine (hint: there should be), there may be more instances of this problem lurking around in the issue tracker.

Proposed solution

In the sam build step, if:

• ${DOCKER_MACHINE_NAME} is defined AND
• docker-machine ssh "${DOCKER_MACHINE_NAME}" uname -r ends with boot2docker

Then sam should perform the following additional steps (or their equivalent through python)

docker-machine ssh ${DOCKER_MACHINE_NAME} \
"mkdir -p \"$(pwd)\" && chown docker.docker \"$(pwd)\""
docker-machine scp --recursive --delta $(pwd) ${DOCKER_MACHINE_NAME}:$(pwd)

By doing these steps manually, I was finally able to run sam local invoke using docker-machine.

/tmp/sam-app$ docker-machine scp --recursive --delta $(pwd) ${DOCKER_MACHINE_NAME}:$(pwd)
sending incremental file list
sam-app/
sam-app/README.md
          4,929 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=24/26)
sam-app/empty.json
              3 100%    2.93kB/s    0:00:00 (xfr#2, to-chk=23/26)
sam-app/issue.md
          1,293 100%    1.23MB/s    0:00:00 (xfr#3, to-chk=22/26)
sam-app/template.yaml
          1,837 100%    1.75MB/s    0:00:00 (xfr#4, to-chk=21/26)
sam-app/.aws-sam/
sam-app/.aws-sam/build/
sam-app/.aws-sam/build/template.yaml
          1,072 100%    1.02MB/s    0:00:00 (xfr#5, to-chk=17/26)
sam-app/.aws-sam/build/HelloWorldFunction/
sam-app/.aws-sam/build/HelloWorldFunction/helloworld/
sam-app/.aws-sam/build/HelloWorldFunction/helloworld/App.class
          2,877 100%    2.74MB/s    0:00:00 (xfr#6, to-chk=13/26)
sam-app/.aws-sam/build/HelloWorldFunction/helloworld/GatewayResponse.class
          1,216 100%    1.16MB/s    0:00:00 (xfr#7, to-chk=12/26)
sam-app/.aws-sam/build/HelloWorldFunction/lib/
sam-app/.aws-sam/build/HelloWorldFunction/lib/aws-lambda-java-core-1.2.0.jar
          7,410 100%    7.07MB/s    0:00:00 (xfr#8, to-chk=11/26)
sam-app/HelloWorldFunction/
sam-app/HelloWorldFunction/pom.xml
          1,509 100%    1.44MB/s    0:00:00 (xfr#9, to-chk=10/26)
sam-app/HelloWorldFunction/src/
sam-app/HelloWorldFunction/src/main/
sam-app/HelloWorldFunction/src/main/java/
sam-app/HelloWorldFunction/src/main/java/helloworld/
sam-app/HelloWorldFunction/src/main/java/helloworld/App.java
          1,399 100%    1.33MB/s    0:00:00 (xfr#10, to-chk=4/26)
sam-app/HelloWorldFunction/src/main/java/helloworld/GatewayResponse.java
            760 100%  742.19kB/s    0:00:00 (xfr#11, to-chk=3/26)
sam-app/HelloWorldFunction/src/test/
sam-app/HelloWorldFunction/src/test/java/
sam-app/HelloWorldFunction/src/test/java/helloworld/
sam-app/HelloWorldFunction/src/test/java/helloworld/AppTest.java
            701 100%  684.57kB/s    0:00:00 (xfr#12, to-chk=0/26)
(sam-java8-helloworld-classnotfound) lestephane@le-studio-xps:/tmp/sam-app$ sam local invoke
2019-07-06 11:13:44 Reading invoke payload from stdin (you can also pass it from file with --event)
{}
2019-07-06 11:13:49 Invoking helloworld.App::handleRequest (java8)

Fetching lambci/lambda:java8 Docker container image......
2019-07-06 11:13:53 Mounting /tmp/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 6c4bafc1-2d84-417f-8edc-8f22077d088c Version: $LATEST
END RequestId: 6c4bafc1-2d84-417f-8edc-8f22077d088c
REPORT RequestId: 6c4bafc1-2d84-417f-8edc-8f22077d088c	Duration: 1852.32 ms	Billed Duration: 1900 ms	Memory Size: 128 MB	Max Memory Used: 5 MB	

{"body":"{ \"message\": \"hello world\", \"location\": \"62.228.125.216\" }","headers":{"X-Custom-Header":"application/json","Content-Type":"application/json"},"statusCode":200}

Caveat: Hot reloads

Hot reloading of code will obviously not work. Basically docker-machine users will have to do sam build everytime they change their code. And that's OK I think.

The following statement in the SAM output will therefore not be applicable

You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically

And so reloading is clearly out of scope.

Why not vboxmanage / virtualbox shared folders?

A workaround that requires the user to have to go modify shared folder settings in VirtualBox, especially when doing a HelloWorld example, adds friction, and still does not ensure that the shared folders behave like real folders, or have the correct permissions. So you end up with more issues from newbies.

In addition, vboxmanage + virtual shared folders are virtualbox specific. docker-machine can also be used with other machine drivers like Hyper-V and VM-Ware.

Why not docker-machine mount?

It mounts a filesystem directory of the docker-machine VM as a local source directory, which is rather useless. You want the Docker VM to mount the user's source code directory, not the other way around. And to this day, docker-machine has no mechanism to achieve that.

Why not NFS, X, Y etc...?

TLDR, as long as a local docker daemon is present, things should work without additional configuration.

A LOT of people are using docker-machine, knowingly or not.

The suggested solution shows how the situation can be improved, if not automatically through sam, at least in a repeatable manual way.

[abhinav-jain09]

@lestephane how do I know the Docker_machine_name varible ?

[lestephane]

@abhinav-jain09 the DOCKER_MACHINE_NAME environment variable is set when evaling the docker-machine env <name> command output

~$ docker-machine create toto
Running pre-create checks...
Creating machine...
...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env toto
~$ docker-machine env toto
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://...:..."
export DOCKER_CERT_PATH="/.../.docker/machine/machines/toto"
export DOCKER_MACHINE_NAME="toto"
# Run this command to configure your shell: 
# eval $(docker-machine env toto)
~$ eval $(docker-machine env toto)
~$ env | grep DOCKER_
DOCKER_HOST=tcp://...:...
DOCKER_MACHINE_NAME=toto
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/.../.docker/machine/machines/toto

[jernejg]

Does does mean you can't use aws sam cli in your Jenkins pipeline if you have Jenkins running in Docker ?

[jfuss]

docker-machine is not support by Docker anymore, making this request not valid.

Closing

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.