Webhook notifications come in 3 times on HA setup

Open mickmister opened this issue 2 years ago • 12 comments

On the community server, which has a 3 node cluster of MM app servers running, I'm getting 3 DM notifications for every operation that sends a notification.

Sep 11 '23 13:09 mickmister

@mickmister Can you please specify the plugin and MM version you are using here?

Sep 18 '23 10:09 raghavaggarwal2308

@raghavaggarwal2308 GitLab plugin 1.7.0 MM latest master, as of submitting the issue

Sep 18 '23 13:09 mickmister

@mickmister Can you please provide us an environment where this issue is getting reproduced?

Sep 25 '23 07:09 raghavaggarwal2308

@raghavaggarwal2308 This is happening on the community server. I'm not sure of any specific reproduction steps. I know we are running a 3 node MM cluster, but I wouldn't expect that to generate 3 posts, since the webhook request only gets handled by one node.

@spirosoik @phoinixgrr I'm curious if you are experiencing this same issue with DM notifications from the GitLab bot

Sep 27 '23 13:09 mickmister

@mickmister No issues from my side. Getting a single notification.

Sep 27 '23 13:09 phoinixgrr

This last occurred for me on 2023-9-11, though that's the only day in a while that I received a notification from the GitLab bot other than the daily reminder

Sep 27 '23 13:09 mickmister

@raghavaggarwal2308 This is happening on the community server. I'm not sure of any specific reproduction steps. I know we are running a 3 node MM cluster, but I wouldn't expect that to generate 3 posts, since the webhook request only gets handled by one node.

@spirosoik @phoinixgrr I'm curious if you are experiencing this same issue with DM notifications from the GitLab bot

yes I have the same issue @mickmister . 3 posts are coming for me. I did a disconnect and removed any redundant subscriptions just to be sure. Though still the same issue.

Sep 27 '23 14:09 spirosoik

🤷 How is this possible? Is there a specific kind of GitLab notifications getting affected?

Sep 27 '23 14:09 phoinixgrr

Confirmed, On this kind of notifications I get multiple:

Sep 29 '23 11:09 phoinixgrr

@spirosoik Any ideas on what we can do to debug this, on the SRE level? i.e. Can we verify there is exactly one request coming in per GitLab user interaction? We haven't seen any similar behavior from other plugins that use webhooks, so the two potential sources of issues seem to be:

Multiple requests coming in
The GitLab plugin is creating multiple/duplicate posts from one incoming request

It very well could be the case that the GitLab plugin is creating duplicate posts, though I would like to rule out the first theory of multiple webhook requests coming in from GitLab. @raghavaggarwal2308 What are your thoughts based on our discussion?

Sep 29 '23 14:09 mickmister

@mickmister feel free to start a thread in SRE channel and the team will act accordingly.

Oct 02 '23 05:10 spirosoik

@mickmister Till now I am unable to reproduce this issue on my side. But I doubt that the plugin code will create multiple posts as I have seen the code and I can't see anything that might get affected by the multi-node environment. The only reason that can cause the issue is that there are multiple subscriptions with the same event in a single channel, but I think @spirosoik has already tested that it's not the case. Also, I have verified that Gitlab is sending only a single webhook request.

Oct 05 '23 11:10 raghavaggarwal2308

@phoinixgrr @spirosoik Are you still facing this issue or we can close it?

Jul 16 '24 12:07 raghavaggarwal2308

@phoinixgrr @spirosoik Are you still facing this issue or we can close it?

Cannot say. Haven't been using the gitlab plugin for sometime now.

Jul 17 '24 16:07 phoinixgrr

Closing for now, can reopen if someone faces the issue in future.

Jul 17 '24 18:07 raghavaggarwal2308