[Feature Request] Queueing a workflow

[baileywickham]

Is your feature request related to a problem? Please describe.

A pain point we have in using Temporal is in queuing workflows. We would like the ability to queue/buffer a workflow directly from that workflow, similar to the executeChild functionality, and without creating a schedule . Currently we are using schedules for this behavior. The schedules do not trigger independently, they are only used with the BUFFER_ONE schedule overlap policy. This has three main disadvantages: the first is we need to generate schedules for workflows which don't really need a schedule, they only need the buffering functionality, the second is we need to create a new client inside each workflow when we queueing the next schedule. Third, you can't pass arguments to the schedule, so you can't have a workflow like RefetchUser(userId).

Describe the solution you'd like

1. The ability to queue a workflow after the completion of a currently running workflow, similar to the BUFFER_ONE schedule overlap policy.
2. The ability to do this from inside a workflow without creating a client, similar to executeChild. It would also be nice to be able to interact with the schedules directly from a workflow without creating a client.

Additional context

Let me know if this is something Temporal would consider implementing and if there is anything I can do to help.

[bergundy]

You could use this pattern to use the workflow itself as a buffer:

// Make sure Request is serializable to JSON (ie. no function, no promises, etc)
type Request = { ... }

// Entity workflow pattern with serialization of request
// (ie. only one request is processed at a time)
export async function myWorkflow(requests: Request[] = []): Promise<void> {
  setHandler(mySignal, (input: Request) => {
    requests.push(input);
    // Don't await here. Otherwise, the Workflow may complete before the promise completes.
  });

  while (!workflowInfo().continueAsNewSuggested) {
		const timeSinceStart = Date.now() - workflowInfo().runStartTime.getTime();
    const shouldProcessMore = await condition(() => requests.length > 0, ms('24h') - timeSinceStart);
    if (!shouldProcessMore) return;

    const request = requests.shift();

    // Process request as appropriate
    await handleSingleRequest(request);
  }

  // Huge histories are bad for performance, so we switch to a new workflow execution whenever
  // history grows over 2000 events. When that happens, we forward any outstanding requests to the
  // next execution.
  await continueAsNew(requests);
}

function handleSingleRequest(request: Request): Promise<void> {
  // It's ok to await here
}

You would then signalWithStart the workflow to add an item to its queue.

[baileywickham]

Thanks for the quick response. This removes the need to create a dedicated schedule and allows us to trigger the next request without an additional client. It does add a fair amount of boilerplate to each of our buffered workflows though, native support would still be ideal. Feel free to close this issue if native support isn't going to be implemented.

Thanks again for the response.