Feature: async invocation for public functions

[alexellis]

Expected Behaviour

Feature: async invocation for public functions

Current Behaviour

Asynchronous invocations are blocked/unavailable.

Possible Solution

A solution would need to include some rate limiting.

A bad actor could potentially carry out a DOS-attack on the asynchronous sub-system or a "queue bomb" through recursion or looping with the X-Callback-URL. We'd need to make sure this was protected against before opening up to the public.

It could be an optional feature for internal clusters for teams?

How would we expose the async routes or modes?

Right now we have:

https://username.domain.tld/function-name/path

How would we specify an asynchronous request? This is needed for anything long-running that also responds to a webhook.

[burtonr]

I was going to suggest an additional header that could be inspected at the router, but with this statement

that also responds to a webhook.

I don't think it's possible to include a header with a webhook. With a webhook, you're limited to the URL right? So, we could add an additional path before the function name https://username.domain.tld/async/function-name/path

Although in that case, what could be done with the rate limiting?

Where would that be handled? Would it be some env var on the gateway/router? Would you need to be able to configure it per function? Could we use an additional label on the function? async-rate-limit: 100

[matipan]

I prefer if we use some form of label. I've seen that knative does something like this with their concurrency label for auto scaling. Is this kind of rate limiting already implemented in the gateway? If not, it would be best if we implemented it there so that regular openfaas users can take advantage of that feature, not only ofc.

The url @burtonr mentioned looks ok to me. The only thing is that users won't be able to have a function called "async" but I don't think that's an issue, "async" is a very vague name for a function. We are gonna have to document that well to avoid confusions if someone decides to use that name. An alternative would be to have something like this: https://username-async.domain.tld/function-name/path and have the edge-router analyze if it ends with -async. But I prefer the one Burton suggested.

[matipan]

I misspoke before. It's not a label what knative uses, it's a field in their CRD spec.

[alexellis]

This doesn't address saturation / resource exhaustion attacks.

[tmiklas]

I would consider a something like this...

Variant 1 - "I really want to shoot myself in the foot":

• introduce config variable into ofc-bootstrap letting user turn NATS on at their own risk... and keep it disabled by default

Variant 2 - "I may want to shoot myself in the foot":

• marker in URL to signal this is to be considered async - I'd propose https://username.async.domain.tld/function to prevent conflicts if async showed up in user or function name
• optional (but recommended) per-function token/API secret set in stack.yml - passed on invocation either as request header or inside POST data... sending token in URL will get it logged at proxies, possibly web trackers, etc.
• router validating token expected vs provided, if marker says this is async, accept or drop (can be 2-stage process - first blindly accept all and then filter what gets passed to function)

Token in varaint 2 needs to be optional because in certain cases it may not be possible to in certain cases to inject it... webhooks from public services or data in XML where router would expect JSON... or simply unable to add token to content.

Best of all, use both variants... Food for thought, nothing more.

[alexellis]

Thanks for your input, this is still a relatively quiet issue, but something that I would like to see more input on.

Option 1 can be achieved by adding the IngressOperator and setting up a FunctionIngress record for each of the async functions.

https://github.com/openfaas-incubator/ingress-operator

Another idea I had was strict rate-limiting on the endpoint, and perhaps banning the X-Callback-URL use.

[tmiklas]

How about...

Add options to stack.yml and functionality that would:

• mark function as async-only (default: false) - would deploy ingress operator automatically for that function
• for async-only functions define X-Callback-URL in stack.yml so it can't be changed at invocation; this would allow sweet building of async pipelines!!!

Now, let's say I want to run cows function in both synchronous and async modes, my stack contains 2 functions that use same image and handler... functions have different, user defined names and one has extra flags as above.

This does not address how to safely authenticate the request, but the more I think about it, the more I think this should be handled by the function, not the gateway. The logic of checking for the token will work as long as function's author codes it. Token can be then either a request header, POST data element or even a digitally signed message with full envelope... does not matter for OFC - it's function's job to handle it.