Execute OpenWhisk action via Ignite and Firecracker VM

[chetanmeh]

This is a proof of concept to execute OpenWhisk actions via Weave Ignite which uses Firecracker MicroVMs

Recently Weave works announced Ignite project which supports running compatible docker images as Firecracker MicroVMs. Such an execution can provide better security compared to simple container based execution.

Objective

Some of the key aspects here are

1. Rebuild OpenWhisk runtime as Ignote compatible docker image
2. Execute action via vm build from those images

Usage

1. Install Ignite

Follow the Ignite installation steps and ensure that required dependencies are met. OpenWhisk would check for ignite cli at /usr/local/bin/ignite

2. Build a Ignite compatible docker image for nodejs.

Currently a docker file is provided for nodejs 12 here

$ cd tools/ignite/nodejs
$ docker build -t whisk/ignite-nodejs-v12:latest  .

3. Build the standalone OpenWhisk

$ ./gradlew :core:standalone:build

4. Start the server

Start the server passing a custom runtime manifest which refers to new nodejs docker image build in previous steps

$ sudo java -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.ignite.IgniteContainerFactoryProvider \
      -jar bin/openwhisk-standalone.jar \
      -m tools/ignite/ignite-runtimes.json 

Note that this is run with sudo as ignite needs root access to run. Once started it would launch 2 pre warm containers (vms!)

In startup logs you can see the vm being created

[2019-07-12T18:12:07.811Z] [INFO] [#tid_sid_unknown] [IgniteClient] Detected ignite client version Ignite version: version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"e0e7e8c50cd3c6532486625393dff4d415081829", GitTreeState:"clean", BuildDate:"2019-07-10T12:30:11Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0
[2019-07-12T18:12:08.080Z] [INFO] [#tid_sid_invokerWarmup] [IgniteClient] running /usr/local/bin/ignite -q image import whisk/ignite-nodejs-v12:latest (timeout: 10 minutes) [marker:invoker_ignite.image_start:1384]
[2019-07-12T18:12:08.101Z] [INFO] [#tid_sid_invokerWarmup] [IgniteClient] running /usr/local/bin/ignite -q run whisk/ignite-nodejs-v12:latest --cpus 1 --memory 256m --size 1GB --name wsk0_1_prewarm_nodejs12 (timeout: 5 minutes) [marker:invoker_ignite.run_start:1406]

Checking running vms

# ignite vm ps 
VM ID                   IMAGE                           KERNEL                                  CREATED SIZE            CPUS    MEMORY          STATE   IPS             PORTS       NAME
296c4f2fd75bf834        whisk/ignite-nodejs-v12:latest  weaveworks/ignite-kernel:4.19.47        17s ago 1024.0 MB       1       256.0 MB        Running 172.17.0.10        wsk0_1_prewarm_nodejs12
e9c77eb3f95e7d7c        whisk/ignite-nodejs-v12:latest  weaveworks/ignite-kernel:4.19.47        17s ago 1024.0 MB       1       256.0 MB        Running 172.17.0.11        wsk0_2_prewarm_nodejs12

There would also be 2 "proxy" docker container for these 2 vm

# docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                                 NAMES
f94d7c15ed90        weaveworks/ignite:v0.4.0              "/usr/local/bin/igni…"   28 seconds ago      Up 24 seconds                                                                                                             ignite-e9c77eb3f95e7d7c
be939a1e2d5a        weaveworks/ignite:v0.4.0              "/usr/local/bin/igni…"   29 seconds ago      Up 25 seconds    

Now you can invoke js action

{
    "namespace": "guest",
    "name": "hello",
    "version": "0.0.1",
    "subject": "guest",
    "activationId": "1b86c507e4df4b6286c507e4df3b62ea",
    "start": 1562937307810,
    "end": 1562937323341,
    "duration": 15531,
    "statusCode": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "LANG": "en_US.UTF-8",
            "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
            "__OW_ACTION_NAME": "/guest/hello",
            "__OW_ACTIVATION_ID": "1b86c507e4df4b6286c507e4df3b62ea",
            "__OW_DEADLINE": "1562937383336",
            "__OW_NAMESPACE": "guest",
            "payload": "hello, world"
        }
    },
    "logs": [],
    "annotations": [
        {
            "key": "path",
            "value": "guest/hello"
        },
        {
            "key": "waitTime",
            "value": 100
        },
        {
            "key": "kind",
            "value": "nodejs:12"
        },
        {
            "key": "timeout",
            "value": false
        },
        {
            "key": "limits",
            "value": {
                "concurrency": 1,
                "logs": 10,
                "memory": 256,
                "timeout": 60000
            }
        },
        {
            "key": "initTime",
            "value": 15526
        }
    ],
    "publish": false
}

Design

This PR adds a new IgniteContainerFactory which internally launches the vm and route calls to it. See Getting started with Ignite to get a quick overview of how ignite works

Upon first launch it would

1. Import the image via ignite image import whisk/ignite-nodejs-v12:latest. This would implicitly pull the image if not present
2. Create and run the vm ignite run whisk/ignite-nodejs-v12:latest --cpus 1 --memory 256m --size 1GB --name wsk0_1_prewarm_nodejs12
3. Connect to the 8080 port of launched vm for actual execution

IgniteContainer would also figure out the matching docker container running for given vm and use that to make calls as being done with standard Docker implementation.

Docker Image

For any docker image to be usable with ignite it must have init system present. For this it provides some base images. This PR uses there centos image to build a custom nodejs action runtime image

FROM openwhisk/action-nodejs-v12:1.14.0-incubating AS ownode

FROM weaveworks/ignite-centos

RUN curl -sL https://rpm.nodesource.com/setup_10.x | bash -
RUN sudo yum -y install nodejs

COPY --from=ownode /nodejsAction /nodejsAction
COPY --from=ownode /node_modules /node_modules
COPY ./action.service /etc/systemd/system/

RUN chmod 664 /etc/systemd/system/action.service \
    && systemctl enable action.service

Service Startup

Despite ignite use of OCI Image the end result is a vm being launched. So for our action runtime to be launches this docker image also adds a systemd service

[Unit]
Description=OpenWhisk Nodejs action server
Wants=network-online.target
After=network-online.target

[Service]
WorkingDirectory=/nodejsAction
ExecStart=/usr/bin/node --expose-gc app.js

[Install]
WantedBy=multi-user.target

So upon startup of vm the node server would be launched and it would bind to 8080 port.

Logging

Currently logs support does not work. For that we need to figure out how to route nodejs logs to vm logs. These logs are then routed to docker logging sub system so we can then fetch it via standard docker log access.

Things to note

Currently the cold start takes long time as for some reason even if vm launches quickly the nodejs server does not respond for some time post start. This aspect would need to be investigated

[sven-lange-last]

@chetanmeh thanks a lot for providing this Ignite / Firecracker starter.

• Would you know how network connectivity is realized for the created VMs?
• Is it possible to run Firecracker VMs within a virtualization like Xen / KVM / VirtualBox or is a bare metal Linux system required?

[chetanmeh]

Would you know how network connectivity is realized for the created VMs

Currently it supports 2 options docker-bridge and cni with docker bridge as default

Is it possible to run Firecracker VMs within a virtualization like Xen / KVM / VirtualBox or is a bare metal Linux system required?

I tested it on bare metal linux (ubuntu) box. Per firecracker docs either a bare-metal machine (with hardware virtualization), or a virtual machine that supports nested virtualization can be used

[130B848]

When I try to start the server by

sudo java -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.ignite.IgniteContainerFactoryProvider \
      -jar bin/openwhisk-standalone.jar \
      -m tools/ignite/ignite-runtimes.json 

Errors occurr in ignite ps command:

[2019-11-21T14:02:11.322Z] [ERROR] [#tid_sid_invoker] [IgniteClient] info: command was unsuccessful, code: 1 (unsuccessful), stdout: , stderr: Error: unknown flag: --no-trunc

And even if I delete these options (--notrunc --format=...), there is still no pre warm containers launched.

Is there anything wrong with my environment configuration?

[chetanmeh]

Have not tried the flow with current ignite version so may be it does not work with current impl and need to be adapted. --no-trunc was used to get full ps output. May be we need to use a different way

[130B848]

Have not tried the flow with current ignite version so may be it does not work with current impl and need to be adapted. --no-trunc was used to get full ps output. May be we need to use a different way

Thanks for the quick reply!

Since I am interested in this PR and want to reproduce this, could you please tell me some more detailed information about the environment configuration with which the ignite works well? (e.g., the version of ignite when you succeeded)

Thanks!

[chetanmeh]

Per logs its 0.4.0

[2019-07-12T18:12:07.811Z] [INFO] [#tid_sid_unknown] [IgniteClient] Detected ignite client version Ignite version: version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"e0e7e8c50cd3c6532486625393dff4d415081829", GitTreeState:"clean", BuildDate:"2019-07-10T12:30:11Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0

[zhuangh7]

Have not tried the flow with current ignite version so may be it does not work with current impl and need to be adapted. --no-trunc was used to get full ps output. May be we need to use a different way

Thanks for the quick reply!

Since I am interested in this PR and want to reproduce this, could you please tell me some more detailed information about the environment configuration with which the ignite works well? (e.g., the version of ignite when you succeeded)

Thanks!

Hi, I am trying to follow the steps here to change openwhis's runtime to firecracker. But I meet the same bug as you. Although I get the specific 0.4.0 version of Ignite, and there is no flags '--no-trunc' or '--format'. Have you solved the problem?

[zhuangh7]

Per logs its 0.4.0

[2019-07-12T18:12:07.811Z] [INFO] [#tid_sid_unknown] [IgniteClient] Detected ignite client version Ignite version: version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"e0e7e8c50cd3c6532486625393dff4d415081829", GitTreeState:"clean", BuildDate:"2019-07-10T12:30:11Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0

Hi, it seems that all versions of Ignite do not have neither '--no-trunc' or '--format' even the given version 0.4.0. Can you give me some pieces of advice?

[chetanmeh]

it seems that all versions of Ignite do not have neither '--no-trunc' or '--format' even the given version 0.4.0.

Strange it indeed does not mention that. May be it used to pass all the flags to docker ps command and that supported those 2 options

[zhuangh7]

it seems that all versions of Ignite do not have neither '--no-trunc' or '--format' even the given version 0.4.0.

Strange it indeed does not mention that. May be it used to pass all the flags to docker ps command and that supported those 2 options

Thank you for your prompt reply, this is a good idea, I will try it tomorrow. Thanks again for your great work.

[lihshyang]

Thanks for the great work! Any idea how long does a cold start take using Weave Ignite?

[rabbah]

Thanks for the great work! Any idea how long does a cold start take using Weave Ignite?

I couldn't find any benchmarks we recorded. Resolving the conflicts on this PR is not difficult, I can do it when I get a chance and you can try it?

[lihshyang]

Thanks for the great work! Any idea how long does a cold start take using Weave Ignite?

I couldn't find any benchmarks we recorded. Resolving the conflicts on this PR is not difficult, I can do it when I get a chance and you can try it?

That'll be great. Thank you!

[style95]

@chetanmeh I marked this PR as stale to close soon due to long inactivity. Please let me know if you would work on it again.