[YSQL] pg_cron tserver service

[xenown]

Jira Link: DB-2758

Description

This is the 2nd change in a series of diffs for porting the Postgres extension pg_cron to yugabyte. The following summary assumes you understand how the original pg cron extension works.

Unlike the original extension, we intend for this port to have all Postgres processes on all nodes act as workers as opposed to a single worker which will complete the scheduled pg cron jobs. One of these workers will also be elected as a leader based on whether the worker is associated with a tserver that is the leader of a designated tablet. This tablet will be the single tablet created for the job table (i.e. the only tablet created from CREATE TABLE job(...) SPLIT INTO 1 TABLETS). The leader will be the worker that is also responsible for assigning job runs to other workers.

The PG Cron extension requires all workers to know what the latest set of scheduled jobs (which is stored in a non-catalog table) is at all times. At the time of writing, Postgres instance A will not know non-catalog table T has been updated should Postgres instance B update T. Thus, A would need to be informed in some way that T was updated. We can accomplish this by signalling A when B has completed a write on T.

Additionally, workers need to know when they have been assigned a new job and leaders need to know when a job is completed. Originally, job assignment was trivial as there was only 1 worker. With multiple workers, we will modify the job_run_details table to also indicate which worker is assigned each job run. Inserting a row into this table implies job run assignment. Updating a row's end_time value implies job completion. Hence, on a successful insert, we signal the worker associated with the new row and on an update to a row's end_time, we signal the leader. The signalling mechanism is similar to above.

We accomplish all this by introducing a Tserver service which will do the following:

• Hook into write RPCs to know when successful writes have occurred on pg cron tables.
• Signal other Tservers of relevant pg cron table modifications
  • Add new RPCs to support this signalling
• Determine the Pg Cron leader and periodically publish the leadership status to the local Postgres and TS shared memory.