semaphore
semaphore copied to clipboard
semaphoreui
Add Support for Webhooks
Add support for Webhooks in Ansible-Semaphore
This MR adds support for Webhooks to trigger Ansible Templates and run Playbooks based on configuration given to a webhook that can be potentially received by semaphore.
Webhook Configuration
The change hereby presented adds a new item on the sidebar titled Webhooks, navigating here will present the user with a list of Webhooks being recognized by the system.
A webhook links to a template in Semaphore which will be run when the e webhook satisfied its extractor configuration and has succeeded all Matcher criteria.
Webhook Extractors
A webhook consists of 1 or more Webhook Extractors which in turn have 1 or more Matchers and Extracted Values
Webhook Matchers
A Matcher defines a characteristic of the webhook event being sent to the semaphore system where the system will read the request for its body or header data and match on fields presented in the webhook.
Consider the following Webhook posted to semaphore system:
POST /webhook HTTP/1.1
Host: semaphore.example.org
Content-Type: application/x-application-json
{
"data": {
"field1": "a",
"field2": "b"
}
}
A matcher may now select to parse the body JSON to retrieve for example data.field1
It may now try to :
- test if a field value is equal
- test if a field value is unequal
- test if a field value contains a string
If all the Matchers for an extractor have been met the system will engage the extractor linked Webhook Extract Values
Extracted Value
An extracted value is any field selected from the webhook to turn into a variable on the template fields/values being passed to the playbook being run as part of the template attached to the webhook object.
Considering the previously described Webhook posted to semaphore system:
POST /webhook HTTP/1.1
Host: semaphore.example.org
Content-Type: application/x-application-json
{
"data": {
"field1": "a",
"field2": "b"
}
}
The extractvalue may now select data.field2 and assign it to VARIABLE_A which would be passed as configuration variable to the ansible-playbook run invoked by the template.
This way the system can augment the received data to provide it to the playbook being run.
Webhook execution
Upon receiving a Webhook that satisfied all matchers and has successfully extracted all the required values into variables, the system will attempt to trigger a run of a template, build or deployment.
@fiftin It would be awesome if you could help me get this over the line as I'm currently hitting some issues with boltdb and updating the objects. I believe part of this is due to the way the buckets are set up?
@fiftin It is finally ready for review.
I will work on the merge conflicts sooner rather than later
Hello,
Do you know when it will be available on the next version?
Thank you.
@fiftin I updated the PR with some UI polish and backend fixes for the boltdb version of it. I'm trying to figure out what's wrong on the CI, if you can help me figure this out, that would be cool.
Here are some screenshots:
Webhooks Main Page
Extractor View
Extractor Actual Configuration
Sorry for stupid question, but could you please clearify for me one thing. As I understood these changes point to webhook to semaphore system, not for an external thing. So, my question is do you have in plans to include external webhook functionality? For situations like after ansible play ends notificate some system in general with data about failure/success, hosts, maybe output optionally? For example, I have some cron-job playbook run and I want to "subscribe" for its runs. I'll be very grateful for the answer Thanks!
Notifications/Pushes/Webhooks from your Ansible Playbooks should be solved differently.
This changeset is not to send webhooks ourselves. I suggest researching means of doing so at the end of your playbooks instead by using another web POST step in them.
On Tue, 9 Jan 2024, 12:09 Alexey Kutsenko, @.***> wrote:
Sorry for stupid question, but could you please clearify for me one thing. As I understood these changes point to webhook to semaphore system, not for an external thing. So, my question is do you have in plans to include external webhook functionality? For situations like after ansible play ends notificate some system in general with data about failure/success, hosts, maybe output optionally? I'll be very grateful for the answer Thanks!
— Reply to this email directly, view it on GitHub https://github.com/ansible-semaphore/semaphore/pull/1365#issuecomment-1882872137, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJEWWH2SNADHLFFQ4O6IGTYNUQODAVCNFSM6AAAAAA2YPPDMOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBSHA3TEMJTG4 . You are receiving this because you were mentioned.Message ID: @.***>
Hi
Has this been completed, really want to use webhooks for my telegram homelab projects
thank you so much for the great work you've been doing
@RustinCassiem Sadly not. I'm struggling to figure out all the problems with the API testing using dredd. As you can see the integration tests are failing. If you could give me a hand I'd be really happy.
Hi @andreas-marschke
I don't understand what this Webhooks do. Is it alternative of Semaphore API? What reason to use Webhooks instead of call Semaphore API?
In the traditional sense, a webhook is a call to external endpoint when certain conditions are met.
I will be very grateful if you provide examples of use.
My goal is to keep the Semaphore as simple as possible - both code and interface. Thanks to this approach, the Semaphore didn't turn into AWX. The complication must provide a clear benefit (for most users) in order to get into the main branch.
Thanks to the simple code, everyone can implement a custom version of Semaphore to suit their needs.
Hope for understanding.
I don't understand what this Webhooks do. Is it alternative of Semaphore API? What reason to use Webhooks instead of call Semaphore API?
In the traditional sense, a webhook is a call to external endpoint when certain conditions are met.
I will be very grateful if you provide examples of use.
@fiftin If you read the Merge Request description, there's an entire explanation of what I need this for, what webhooks are supposed to do and why it is beneficial to support them.
There's also the description I dumped into issue #1319, that you yourself had acknowledged and set "feature" as a flag on.
The intent is to have a generic implementation to do EXACTLY not much more than the product is supposed to.
My goal is to keep the Semaphore as simple as possible - both code and interface. Thanks to this approach, the Semaphore didn't turn into AWX. The complication must provide a clear benefit (for most users) in order to get into the main branch.
Thanks to the simple code, everyone can implement a custom version of Semaphore to suit their needs.
Hope for understanding. @fiftin I do not intend to make Semaphore into AWX, I was merely seeking to enhance Semaphore with the ability to work as a inter-application gateway to trigger playbook runs allowing us to make simple tasks such as deployment of keys, updates to services, provisioning based on CI/CD/VCS events. Webhooks are a common industry tool to make product/service intercom on a push-notification-style basis easier.
This does not complicate but enhance the value of the simplicity of the product that semaphore is.
Notifications/Pushes/Webhooks from your Ansible Playbooks should be solved differently. This changeset is not to send webhooks ourselves. I suggest researching means of doing so at the end of your playbooks instead by using another web POST step in them.
I planning to add support Bash and other runners not just Ansible Playbook. So call external endpoints from Semaphore is a good idea ;)
Endpoint /endpoint should be joined to project like /projects/1/integrations/endpoint because Semaphore can have a lot of users and projects (for example we have Semaphore Cloud). So GetAllIntegrationExtractors can return a very big array.
Also we should add request verification by token or secret:
- for GitHub: https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
- for GitLab: https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#validate-payloads-by-using-a-secret-token
Endpoint
/endpointshould be joined to project like/projects/1/integrations/endpointbecause Semaphore can have a lot of users and projects (for example we have Semaphore Cloud). SoGetAllIntegrationExtractorscan return a very big array.
The cloud offering wasn't in scope for me. Thanks for pointing that out. The projects/ subpath wouldn't require authentication though, correct?
Also we should add request verification by token or secret:
- for GitHub: https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
- for GitLab: https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#validate-payloads-by-using-a-secret-token
This was what I wanted to achieve with matchers once extended in its implementation. For now only checking on (un)equals/contains
The cloud offering wasn't in scope for me. Thanks for pointing that out. The projects/ subpath wouldn't require authentication though, correct?
Yes, it is correct. Just URL will be /project/111/integration/endpoint instead of /integration/endpoint.
@fiftin Do you want to also do the UI Update to expose the AuthMethod in the Integration? Otherwise I'd take a stab at that, while it would be good to get the test-integration resolved.
Notifications/Pushes/Webhooks from your Ansible Playbooks should be solved differently. This changeset is not to send webhooks ourselves. I suggest researching means of doing so at the end of your playbooks instead by using another web POST step in them.
I planning to add support Bash and other runners not just Ansible Playbook. So call external endpoints from Semaphore is a good idea ;)
this is actually the thing I asked in this tread before. so I have in plans to start making it soon. Hope this would be made as soon as possible and will be handy
@fiftin Thank you for the support and help getting this over the line. When do you plan on having the next release? I'm fine with a Beta so long as I have a container to run.
It's available in the develop branch tag on docker but has a bug for
which a patch is still to be merged. See: !1816
is this feature available in the docker container yet?
