Support outgoing webhooks

[paulfitz]

Grist should allow the user (and integrators) to trigger other services on common events such as record updates.

Without webhooks, users (and integrators) need to resort to polling, which is less than ideal for everyone concerned. Our Zapier integration currently uses polling, but if we had webhooks it could be upgraded to 'instant' triggers, meaning faster workflows for users and less server load for us.

Update: there is some unofficial support for webhooks, for those willing to use the Grist API to configure them. For hosted Grist, Webhook URLs are allowed for the following domains:

• [x] zapier.com
• [x] pabbly.com
• [x] webhook.site
• [x] meta-api.io
• [x] pipedream.com, pipedream.net (from 8/1/2022)

If you need a different domain, please request it. If self-hosting, be sure to add the domain you care about to a comma-separated ALLOWED_WEBHOOK_DOMAINS environment variable (e.g. zapier.com,pabbly.com,webhook.site).

[paulfitz]

This is getting closer. If anyone would be interested in being a beta tester of webhooks (and particularly Zapier instant triggers), we'd like to hear from you.

[HermesRatgers]

This is getting closer. If anyone would be interested in being a beta tester of webhooks (and particularly Zapier instant triggers), we'd like to hear from you.

I am interested, but mainly for Pabbly Connect.

[paulfitz]

Great, thanks @HermesRatgers. We'll add Pabbly Connect to the domains allowed for webhooks, and prepare instructions for how to get this set up. Are you comfortable using Grist's API (https://support.getgrist.com/rest-api/)? One of the steps will involve making a manual API call, since we don't have a way to edit webhooks visually in the app yet.

[HermesRatgers]

I know how to make api calls (I even know how to create integrations from within Pabbly)

[paulfitz]

Here is a sketch of how to use webhooks with Grist. The procedure isn't as smooth as I'd like, which is why I'm writing it in a github comment rather than our support site:

• If using hosted Grist, check that the webhook you need is for a domain listed in https://github.com/gristlabs/grist-core/issues/76#issue-972791347 - if it is not, ask Grist staff to add the domain. If self-hosting, follow the instructions in that comment for setting ALLOWED_WEBHOOK_DOMAINS.

• Make a practice Grist document that has a table called Stuff, a text column called Name, and a toggle column called Ready. Make a few rows. Make sure all Ready cells are unset/blank/off - it will represent whether a row is available for sending to the webhook. You can read about readiness columns here: https://support.getgrist.com/integrators/#readiness-column. They are optional, but make testing simpler. All table and column names are arbitrary and have no special significance.

• Get comfortable using the Grist API, documented at https://support.getgrist.com/rest-api/, e.g. do some exercises to add rows to that table (be sure to replace XXXX with your api key and DDDD with the document identifier in the document’s URL, and the http://docs.getgrist.com part if self-hosting):

curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
  https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/records \
  -XPOST \
  -d '{"records": [{"fields": {"Name": "Test"}}]}'

• Practice adding and removing a dummy webhook via the API. A handy place to get one is webhook.site, you can visit there and click “copy to clipboard” beside “Your unique URL”.
• Here’s how to add a webhook, using the unofficial _subscribe endpoint. Be sure to replace my example webhook.site url with your own one:

curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
  https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/_subscribe \
  -XPOST \
  -d '{"eventTypes": ["add", "update"], "url": "https://webhook.site/f02fdeea-6b67-4a03-932a-1a13f8d9e020", "isReadyColumn": "Ready"}'

• The eventTypes list is something you can tweak - include add to get new rows (once they are marked ready), and update to get changed rows, and both to get both kinds of event.
• You should get back a result with some important keys, like this:

{"unsubscribeKey":"60fe7777-d7f8-4521-8e77-b6a9f93b0588","triggerId":1,"webhookId":"f717d5ce-5c26-4d3b-ac4e-49b5dbd93761"}

• These are important to keep track of, because you need them to remove a webhook. There’s no way currently to remove a webhook without them.
• Now, toggle a cell in the Ready column to ON for one of the rows in the document. You should hopefully see something arrive on your webhook.site page! Something like:

[
  {
    "id": 6,
    "manualSort": 6,
    "Ready": true,
    "Name": "Miguel de Cervantes"
  }
]

• It takes a long time for information sent to a bad webhook to be abandoned, if anything goes wrong, you can reset the whole process and clear any faulty requests that might be queued for a document by calling:

curl -H "Authorization: Bearer XXXX" -XDELETE https://docs.getgrist.com/api/docs/DDDD/webhooks/queue

• Go ahead and practice removing the webhook by sending the unofficial _unsubscribe endpoint exactly what _subscribe returned earlier:

curl -H "Authorization: Bearer XXXX" -H "Content-Type: application/json" \
  https://docs.getgrist.com/api/docs/DDDD/tables/Stuff/_unsubscribe \
  -XPOST \
  -d '{"unsubscribeKey":"60fe7777-d7f8-4521-8e77-b6a9f93b0588","triggerId":1,"webhookId":"f717d5ce-5c26-4d3b-ac4e-49b5dbd93761"}'

• You should get a result with "success": true in it back. Okay! After that practice, we’re ready to try the real thing.
• Log in to Pabbly Connect, click Create Workflow, name the workflow, then click the Webhook trigger.
• Click Copy to get the Webhook URL in your clipboard. It should look like:

https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXXXXX (https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXX_3D)

• Make sure to set the Simple Response toggle to OFF! Grist will send the data with some nested structure that could otherwise confuse Pabbly.
• Now, repeat the _subscribe call we did earlier, but with the URL replaced with the real thing (https://connect.pabbly.com/workflow/sendwebhookdata/XXXXXXX_3D).
• Keep the API result you get in case you want to later call _unsubscribe.
• Now, toggle a cell in the Ready column to ON for one of the rows in the document.
• Go back to Pabbly, and you should see that information arrived! You can go ahead and map it using the usual Pabbly process.

@HermesRatgers I'd be interested to hear how this goes for you, and if you hit any blockers or points of confusion.

[HermesRatgers]

@paulfitz Added it and it works, I also added this thread to Pabbly so they can add the webhook aswell. Btw the integration can be managed by you aswell, simply request this by email them your pabbly emailadres and request editor permission. https://pabbly.hellonext.co/b/Update-Existing-Application/p/webhook-triggers-for-getgrist

[HermesRatgers]

@paulfitz Can the date be changed? In webhook I receive a date as Birthday | 494726400 in stead of 1985-09-05

[paulfitz]

@HermesRatgers if something isn't formatted as you like, one option could be to add another formula column that reformats it. For example, if you have a column called Date with dates in it, you could make another column called (for example) DateString that is just =str($Date). Would that work for you? There's some more information on formatting dates on the support site https://support.getgrist.com/dates/#additional-resources

[HermesRatgers]

@paulfitz No, thats not it. The date is Grist is formatted and displayed correct as 1985-09-05. But in the webhook its formatted as 494726400.

[paulfitz]

Grist stores dates as seconds since the Unix epoch (https://en.wikipedia.org/wiki/Unix_time), and then formats and displays them as the user wishes in the UI. With webhooks, you are currently getting the data as Grist stores it. There's no way to control that right now, so I'm wondering if a workaround might be to add a second column that calculates a value that has the formatting you want applied to it?

We're looking at giving more control over how data is rendered across our APIs, and for webhooks.

[alexmojaki]

Just to add some extra clarification, a column of type Date will always return a number like that no matter what Date Format you set in the column options, but a formula column of type Text or returning a Python string as in the suggestions will store the same string and return it in the API so it'll match what you see in the UI.

[tonyallan]

Give me /events, not webhooks is interesting and may be easier to implement and support.

[simplynail]

Hi, is there any ETA for the webhooks feature coming in?

[paulfitz]

@simplynail do you mean a UI for creating webhooks? No, there's no ETA for that, sorry. We do want to get it done when we have capacity, I know that isn't very helpful for you.

[simplynail]

Hi, yes, I meant UI actually. I just was wondering if it's still on your radar and perhaps coming soon. Thanks

[fabiomoratti]

I'd like to propose (hope this is the right place) two new domains to be added to the ALLOWED_WEBHOOK_DOMAINS list:

• amazonaws.com
• on.aws

this will allow to configure webhooks that can reach AWS API Gateway and Lambda Function URLs endpoints, allowing developers (possibly many of them) to easily test custom integrations.

For reference:

• Lambda Function URL structure is: https://<url-id>.lambda-url.<region>.on.aws
• API GW URS structure is: https://<api_id>.execute-api.<region>.amazonaws.com

See here, here and here for the official AWS documentation.

Thanks

[paulfitz]

Hi @fabiomoratti, this is the right place to ask. Normally we can can add domains to an allow list pretty quickly. But access to AWS endpoints in particular may need to wait for a little refactor before we're comfortable that it doesn't create a new kind of attack surface when Grist itself is hosted on AWS infrastructure. Sorry for the delay.

[fabiomoratti]

@paulfitz , thank you for your reply.

In a way, amazonaws.com and on.aws may be considered "general-purpose" domains useful for all the users that deploy their webhooks on AWS, as an alternative it should be possible, on my side, to configure a custom domain for the AWS API GW / Lambda and add it, on your side, to the ALLOWED_WEBHOOK_DOMAINS; of course this solution does not scale because it works for one customer (me, in this case) and on the long run it's not reasonable for you to maintain a long list of allowed domains. Given the above, what is your sentiment about addirng a custom domain while the refactoring is being implemented?

As for alternatives I understand that the way to go is polling: do you have any suggestion for a reasonable polling interval that does not disturb Grist's infrastructure? API calls are limited to 5000 per day per document so one call per minute is well below the limit, but it's worth checking.

Thank you in advance for your support.

[acidturtle]

Hello, thanks very much for making Grist. It's been very easy to work with and I just found out about the support for outgoing webhooks. This is a really nice feature.

I was wondering if you could add netlify (api.netlify.com) to the list of supported domains?

Thanks.

[paulfitz]

@acidturtle I added support for netlify.com to the queue, should take effect on November 28, 2022.

@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).

[acidturtle]

@paulfitz: thank you for the quick response and for adding netlify.com, much appreciated.

[berhalak]

• Prepare to do a practice run on a fresh, disposable Grist document, rather than the document you ultimately intend to use. This is because it can take a long time for information sent to a bad webhook to be abandoned, and there’s no easy way yet to reset that process. So it is better to practice on something low-stakes.

There is now a new API endpoint (still unofficial) that can reset the whole process and clear any queued requests for a document.

curl -H "Authorization: Bearer XXXX" -XDELETE https://docs.getgrist.com/api/docs/DDDD/webhooks/queue

[berhalak]

We added a new endpoint /webhooks. It returns some basic configuration and monitoring information about all webhooks registered in a document. It is available currently only for document owners. You can query it using:

curl -H "Authorization: Bearer XXXX" -XGET https://docs.getgrist.com/api/docs/DDDD/webhooks

It returns a JSON with a following schema:

{
   webhooks: [
    {
      id: string, // webhook id
      fields: {
        url: string,
        unsubscribeKey: string,
        eventTypes: string[],
        isReadyColumn?: string|null,
        enabled: boolean,
        tableId: string,
      },
      usage: {
        lastEventBatch?: { // last attempt information
          status: 'success'|'failure'|'rejected',
          httpStatus: number,
          errorMessage: string | null,
          size: number, // number of requests in the batch
          attempts: number, // number of attempts
        },
        lastSuccessTime?: number, // last time the webhook request succeeded
        lastFailureTime?: number,
        lastErrorMessage?: string | null, // last error message and HTTP status
        lastHttpStatus?: number|null,
        updatedTime?: number|null,
        numWaiting: int, // how many requests are queued currently
        status: 'idle'|'sending'|'retrying'|'postponed'|'error'
      }
    }
  ]
}

Webhook status description:

• idle - there are no active requests
• sending - there is an active request
• retrying - there is an active request, but the last attempt failed
• postponed - all attempts have failed, webhook was added to the end of the queue
• error - all attempts have failed, but the queue is too long, so some requests were dropped. lastEventBatch will have rejected status.

We also modified the /_unsubscribe endpoint. Now, it only requires knowledge of the webhook id (if it is called by the owner) or the webhook id and unsubscribeKey otherwise. Owners can query both these keys using the new /webhooks endpoint, so there is no need to memorize them anymore.

[fabiomoratti]

@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).

@paulfitz: thank you for your support; I've been playing both with Grist APIs and webhooks and I'm confident I can make it work so I'll write to the mail you mentioned to have my custom domain added waiting for the refactor. Additionally if you feel it may be useful/interesting for other people I'm willing share the code (a PoC-like version) I use to implement the AWS Lambda webhook; I was going for a public GitHub repo but if you have a better place just let me know.

[paulfitz]

@fabiomoratti we could add your custom domain, as a stop-gap (write [email protected] and mention my name if you don't want to post it here).

@paulfitz: thank you for your support; I've been playing both with Grist APIs and webhooks and I'm confident I can make it work so I'll write to the mail you mentioned to have my custom domain added waiting for the refactor. Additionally if you feel it may be useful/interesting for other people I'm willing share the code (a PoC-like version) I use to implement the AWS Lambda webhook; I was going for a public GitHub repo but if you have a better place just let me know.

Your custom domain will be supported from Monday. Sharing your PoC in a public github repo sounds great, thanks @fabiomoratti !

[paulfitz]

Update: webhooks are available from the UI now https://support.getgrist.com/newsletters/2023-05/#webhooks

There is no longer any restriction on the domains our hosted service will deliver to. Self-hosted Grist can be configured similarly by setting ALLOWED_WEBHOOK_DOMAINS to * but, depending on your trust model, you may either want to:

• Stick with explicitly allowing certain domains, or
• Configure a secure proxy (and tell Grist to use it with GRIST_HTTPS_PROXY) to deliver webhook payloads.

A threat to watch out for is accidentally allowing untrusted users to poke around at your internal endpoints.