feature request: JobSink Integration / Long Running Functions

[philipbjorge]

Usecase

We are evaluating using KNative in our organization and one of our usecases are long-running event handlers -- Think 5-120 minutes. We would like to use KNative Functions but from reading the documentations we believe we would run into trouble with such usecases.

JobSink

We saw the JobSink capability in the Event system, but it does not appear to cleanly integrate with the knative functions concept.

(We would have to produce our own docker images and surrounding runtime to forward in the mounted event to our handler.)

Dev Notes

I reached out to @pierDipi who implemented the JobSink and they had the following to share:

I think we have a few options, what I'm leaning towards is:

• JobSink injects an env variable like KNATIVE_EXECUTION_MODE: batch
• when KNATIVE_EXECUTION_MODE: batch various func runtimes would read the event from the file and pass it to the handler
• If the function returns a new event it would be produced to an endpoint that is pointed by K_SINK env variable (so that you can use JobSink in combination with SinkBinding to have "job callbacks")

https://cloud-native.slack.com/archives/C04LGHDR9K7/p1732225883467409?thread_ts=1732216204.039569&channel=C04LGHDR9K7&message_ts=1732225883.467409

I'm very new to this platform, but the above should work well for our usecase and seems to fit well with the existing platform conventions.

[matejvasek]

@pierDipi @lkingland Do we need to do integration in each language/runtime? Couldn't we just include some sidecar that would do the translation?

[matejvasek]

@pierDipi @lkingland @matzew WDYT ^^^ ?

[lkingland]

That might work. I'd love to hear your thoughts on how this would work. Would you mind writing it up, or we can discuss the solution on the next functions meeting.

[pierDipi]

As pointed out in the meeting, the env variable name is called K_EXECUTION_MODE now, to be consistent with K_SINK, etc

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.