Task Executors in Other Languages

[jcgsville]

Intro

It is not practical to use Node.js for all asynchronous work. In green field development, machine learning and other data-intensive or compute-intensive applications often mandate the use of other languages. Some teams like mine may also need to interact with C# APIs and other aspects of the Microsoft ecosystem. We would like to use Node.js for most task executors related to business logic and web application development, but selectively implement some task executors in other languages.

Another use case would be using Graphile Worker without any usage of Node.js. In my opinion, the most powerful aspects of Graphile Worker are 1 - the ability to enqueue a job within the same transaction as writing to your application data store and 2 - Postgres's LISTEN/NOTIFY starting jobs in milliseconds. My understanding is that neither of these super powers require Node.js, and expanding Graphile Worker to other language ecosystems would allow many more engineers to leverage Postgres as a job queue.

Additionally, I can imagine many companies and teams may have legacy code written in other languages that would be impractical to rewrite in Node.js. Supporting task execution in other languages may provide those teams with an on-ramp to move to Graphile Worker more gradually.

Today, Graphile Worker supports task executors in other languages via loading executable files. This works well if there is little startup cost. However, some languages and some specific use cases may have a large startup cost. Examples would be loading a large python project, or loading a large parameters file from disk for a deep learning task.

In an ideal world with infinite developer capacity, it would be cool to see versions of Graphile Worker in many popular languages that are identical at the postgres layer so that people could easily use one language or several languages interchangeably with the same queue store. Regardless of the existence of versions of Graphile Worker in other languages, I think it makes sense for today's Graphile Worker to add support for task executors in arbitrary languages by communicating over unix sockets with local servers containing task executors.

Technical Proposal

I propose we add an opt-in feature to Graphile Worker that allows local servers to host task executors to process jobs. For the purposes of this proposal, I will call these local servers "sidecar servers" to distinguish them from Graphile Worker's primary Node.js process. Feel free to suggest alternative names 🙂

I have spent some time trying to understand how Graphile Worker's internals work to propose a set of changes. Feel free to suggest many changes to my proposal or an entirely different approach if I missed or misunderstood some aspects of Graphile Worker 🙂.

LoadTaskFromSidecarServerPlugin.ts

Most of the changes to Graphile Worker will be in the form of this new plugin which will not be included in the default preset.

This plugin will add a hook called loadTasksFromSidecarServer. This hook will send an initial message to the sidecar server requesting a list of task identifiers that it supports. The sidecar server will respond with a message with a list of task identifiers that it supports. Similar to LoadTaskFromExecutableFilePlugin, loadTasksFromSidecarServer will make a task function for each identifier. This task function will send the full job information over the socket to the sidecar server. When the sidecar server responds with a message indicating it has completed the job, the function will resolve. This plugin will use a single shared connection to the sidecar server.

The plugin will need to be configured with a file path at which it will open a socket for communication to and from the sidecar server. Both the Graphile Worker Node.js process and the sidecar server must have permissions to read and write from that file. I am not exactly sure how the file path will be configured in graphile worker, as I'm not yet deeply familiar with Graphile Config. I think when the plugin is in use, it can pull from a property like worker.sidecarServerSocketPath or something like that.

The mechanism by which the runner and CLI gets tasks will need to be modified to support the new option, as the current mechanisms are pretty tightly coupled to the notion of a task directory. We could either add a condition for this new mechanism or we could create a bit more of a decoupled interface so that future task loaders can be added with fewer changes to CLI and runner. I imagine that decoupled interface would look like changing task loading to a single loadTasks hook that each plugin can implement.

The Sidecar Server

The sidecar server will be very simple. It needs to listen to the socket at the same path as configured in the Graphile Worker Node.js process. Its primary function will be to route jobs to calls to task executors according to the task identifier of the job. The sidecar will respond on the same socket with a success message when a job is complete.

The messages sent between the Graphile Worker Node.js process and the sidecar server will be serialized as JSON.

There may need to be some heart beat to ensure that the connection between to two processes has not broken. I don't think this is necessary, though.

Breaking changes

By being opt-in, this feature will introduce no breaking changes to Graphile Worker. There will need to be some changes to cli.ts and runner.ts which will change code on the critical execution path for all Graphile Worker users. These changes should not introduce any incompatibility for existing users of Graphile Worker. If we desire to minimize or eliminate changes to CLI or runner, we could pursue a different implementation that relies on a task directory, or at least a simulated task directory.

Supporting development

I:

• [x] am interested in building this feature myself
• [x] am interested in collaborating on building this feature
• [x] am willing to help testing this feature before it's released
• [x] am willing to write a test-driven test suite for this feature (before it exists)
• [x] am a Graphile sponsor ❤️
• [ ] have an active support or consultancy contract with Graphile