gitea [Proposal] Notify runners to fetch Actions tasks instead of polling

Background

Act runner connects to the Gitea instance via short HTTP connections. So runners have to ask for new task again and again.

Gitea will start a transaction to find and assign a task to the runner when it requests a new one. However, we know that there may not be a task available most of the time, so Gitea has to roll back the transaction and respond with "no task yet, try again later."

That’s why you can see there are so many POST /api/actions/runner.v1.RunnerService/FetchTask in router logs, and [SQL] ROLLBACK [] in SQL logs.

It doesn't hurt but it's annoying. And it wastes network and database resources to some extent.

Solution

Provides a mechanism for Gitea to notify runners

We need a way to notify runners from the Gitea side. Here are some possible ways:

gRPC Streaming. Not sure if it works with gRPC over HTTP.
WebSocket. Actually, it was used in earlier versions of Gitea Actions.
Blocking Queries. It's the way in which the consul watches changes.
Server-sent events.

I'm not sure which approach would be best. However, one thing is certain: the approach must be compatible with the proxy. This is because there is always an HTTP proxy in front of Gitea, such as Nginx. IIRC, Nginx with default config closes connections that have been inactive for a long time.

Reuse old logic to fetch task

To be clear, Gitea will notify runners only with a simple message "there may be a new task", instead of the whole task data.

So it could be easier to implement, whenever the relevant data table changes, Gitea could notify all runners. The runners will then request new tasks, just as they do now. If they don't get a new task, maybe the label does’t match, they can just continue to wait to be notified again.

On the other hand, runners may not immediately request new tasks when notified because they may already be at capacity, and they will request later when they are idle.

Therefore, forcefully sending tasks to runners without their request may not be feasible or may be very complicated.

May 05 '23 09:05 wolfogre

Hmm… Yes, as long as that is entirely optional I agree that something like that can be useful. I'd object to making it required, for example for the following use case: You only have a runner inside your intranet, and pull tasks from a public Gitea instance. The only thing I'm thinking about is whether it improves the situation so much that it is worth the effort. What is the default timeout in act_runner? 5 seconds?

May 05 '23 10:05 delvh

... I'd object to making it required, ...

Sure, it's optional. Runners can skip the wait and go straight to regular polling when:

Gitea or reverse proxy does't support it.
It's disabled on Gitea side.
It's disabled on runner side.

... whether it improves the situation so much that it is worth the effort.

Exactly. That's why this is a proposal issue, not a PR.

... What is the default timeout in act_runner? 5 seconds?

5 seconds for timeout, 2 seconds for interval.

May 05 '23 10:05 wolfogre

I feel Blocking Queries, aka long polling would be a natural way to go. This is also how GitHub self-hosted runner do it.

The self-hosted runner uses an HTTPS long poll that opens a connection to GitHub for 50 seconds

Nginx has a timeout of 60s by default. So 50s should work with no configuration required. Maybe Gitea can attempt to poll by default, and then fall back to regular polling if received several 524 Gateway Timeout error.