Custom Lambda runtime?

[armanbilge]

Following from https://github.com/typelevel/feral/issues/132, @baccata points out that AWS offers a similarly low level interface to build a custom Lambda Runtime.

https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

The advantage of implementing a custom runtime is the capability to process multiple events concurrently, but it's unclear if AWS supports this. On the one hand, polling for incoming events is completely within the lambda's control, and an id is used to pair responses with events so they don't necessarily need to be processed in order. But on the other, there are some strange assumptions such as the use of environment variables to pass around tracing headers, which makes me think this is unsupported behavior.

This should be further investigated.

[bpholt]

I was wondering if a custom runtime would let us more completely manage resource lifecycles (so that it could run the finalizers when the runtime shuts down, instead of skipping them like it does now). I'm not sure if that's actually supported or not either.

[armanbilge]

That's a good idea, that one has bothered me as well. I don't see anything in the custom runtime docs about a shutdown phase, but I did look into this at least a couple times: there is a "Lambda Extensions" API (do you know anything about this?) that does get a shutdown hook:

Shutdown: This phase is triggered if the Lambda function does not receive any invocations for a period of time. In the Shutdown phase, Lambda shuts down the runtime, alerts the extensions to let them stop cleanly, and then removes the environment. Lambda sends a Shutdown event to each extension, which tells the extension that the environment is about to be shut down.

https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html

It's frustrating if the extension has access to this important lifecycle event, that the lambda itself does not.

[armanbilge]

Not official by any means, but I found this:

https://github.com/RhinoSecurityLabs/Cloud-Security-Research/blob/b7d14b7eb51b7dea6ea6a8978682bd9671e0e183/AWS/lambda_ssrf/README.md

AWS Lambda functions expose an internal API on localhost port 9001 that is meant for custom Lambda runtimes, but it is accessible from within an active invocation of a function. This means that if you get remote code execution or a server side request forgery vulnerability within a Lambda function, you can query that API. The /2018-06-01/runtime/invocation/next endpoint will return the event data that was passed into the function for that current execution, where you may be able to find secrets or other useful information to extend your attack into the environment.

The key words there being current execution. This suggests that until the response is given for the event in hand, that the endpoint will always return the same event (and not the next one). It is also notable that it is a GET endpoint, so there should be no side-effect to calling it.

[Baccata]

I suppose it makes sense, considering how function calls will be retried in the event of an exception.

Oh well 🤷‍♂️

[armanbilge]

So new thoughts on this: now that Scala Native support is slowly rippling through, would be super awesome to have Feral Native. This would necessitate writing our own Lambda runtime.

And in fact, said runtime would not have to be specific to Scala Native: it could be pure code that cross-compiles for all three platforms. Deploying a Lambda with a custom runtime is certainly more difficult, but it can also offer several advantages since we'd control all the threads and be able to allocate global resources without doing weird unsafe stuff.

This suggests we could benefit from an abstraction that allows the runtime to be swapped out on any platform, e.g. for the AWS JVM runtime or our own runtime.

[armanbilge]

Oh dear, I'm repeating myself 😂 https://github.com/typelevel/feral/issues/209#issuecomment-1175678334

[armanbilge]

I've pushed a branch that shuffles the repo to setup for this. https://github.com/typelevel/feral/tree/feature/custom-lambda-runtime

It:

1. Extracts a lambda-kernel, with Context, Events, and TracedHandler, but no IOLambda.
2. The http4s and cloud formation integrations are now based on the kernel.
3. Creates a lambda-runtime, which depends on kernel. This is where the runtime implementation will live. https://github.com/typelevel/feral/blob/26b16229a9abd70b9cd7832b1cd58946aebc5cb7/feral-lambda-runtime/src/main/scala/feral/lambda/runtime/LambdaRuntime.scala#L25-L35
4. Adds Native cross-builds.

The idea is that:

1. The core lambda module will eventually become an opinionated module with the AWS Java runtime for JVM, the AWS Node.js runtime for JS, and an Ember-client-based Feral runtime for Native (probably backed by epollcat, or maybe fs2-io_uring). This is how I expect it to be most commonly used on each platform.
2. Anyone that wants to use a different HTTP client on Native, or use the Feral runtime on JVM or JS, or even use a different effect type, can just do a custom thing with the feral-lambda-kernel and feral-lambda-runtime modules.