HTTP Webhook for sync jobs

[mri-dula]

Problem

I would like to write custom jobs/workers, that extract information from the synced destination and save it into a different database of my own. However, there is no way of knowing when a particular sync job is completed. The only alternative is scheduling my job to run every x minutes, but I don't want to do that.

Solution

I saw that there is something called post_handle_destinations which can be used to fire a DBT Webhook every time a sync job is completed. Maybe we can expand it to fire generic HTTP Webhooks (instead of keeping it limited to DBT).

The information that would be useful to this Webhook would be:

1. source_id
2. destination_name
3. A boolean - were there any writes done?
4. source_type

There was some discussion on this topic on Slack, linking it here.

[absorbb]

@mri-dula I've made a simple fix that allows any destination to work as post_handle_destinations (without UI support yet) But without boolean - were there any writes done - that will require more than a simple fix. Please tell if that is critical for you.

Event format is:

{
  "_timestamp": "2022-04-20T08:40:03.195255Z",
  "destination_ids": [
    "postges1"
  ],
  "event_type": "SOURCE_SUCCESSFUL_RUN",
  "source": "redis1",
  "source_type": "redis",
  "status": "SUCCESS"
}

Source config:

sources:
  redis1:
    type: redis
    destinations:
      - postges1
    post_handle_destinations:
      - my_webhook
    collections:
      ...

Fix is a part of the beta image now. You can test with switching to the beta docker image tag.

[mri-dula]

@absorbb Thank you so much! This has been of great help!