Webhook failed with error code 403 caused by hmac validation

[emmajxli]

Issue summary

A partner submitted a ticket for the issue for one of their orders/fulfilled webhook. This webhook usually get 100 orders fulfilled everyday within 1 or 2 hours. Most of the orders work properly, but a small number of orders failing with an error code of 403.

The error message is: Could not validate request for topic orders/fulfilled. In the last 4h, 8.9% of the webhook delivered returns this 403 error.

Looking at the code that returns this error, it can be seen that this is due to hmac validation failing: https://github.com/Shopify/shopify-node-api/blob/e451ab91e7e64ae191c63ff953eb3f0e88431d2a/src/webhooks/registry.ts#L387-L416

However, the hmac header value originates from Shopify when calling our webhook - hence this validation should never fail. So either the calling side or the validation side is not 100% bulletproof - in other words for certain orders, the code is not working correctly.

Expected behavior

The error should not show up.

Actual behavior

I had a discussion with our webhook developer Marcus, and he confirmed that he has chatted with the partner, the webhook failed to validate 11 times, and then succeed on the 12th attempt. He also checked and discussed this issue with team but still cannot has the answer for why this would happened.

Other useful resource:

Shopify community post for this post Ticket submitted by the partner

If any additional info needed, please contact me via slack: Emma Li

Steps to reproduce the problem

N/A

Reduced test case

N/A

Checklist

• [x] I have described this issue in a way that is actionable (if possible)

[mariusa]

Same here (response 401 with 5.0.1 version). See also #320 , #275 Changing 1 character in product description (adding a space) fixes the validation, removing the spaces triggers it again.

[haveneersrobin]

Are there any updates regarding this? I am getting 403s for my application in 'some' shops and for 'some' webhook calls.. So it is very much not consistent. This is quite frustrating as our webhooks are now getting deleted.

Could this have to do with too many requests in a short period of time? Would batching/queueing them help?

cc @thecodepixi

[mariusa]

Could this have to do with too many requests in a short period of time? Would batching/queueing them help?

No, this always happens on some products, in some cases. I was able to replicate, but ignored by Shopify. There's is an issue with Shopify validation code.

[haveneersrobin]

Could this have to do with too many requests in a short period of time? Would batching/queueing them help?

No, this always happens on some products, in some cases. I was able to replicate, but ignored by Shopify. There's is an issue with Shopify validation code.

@mariusa I've been in contact with Shopify support regarding this. For us the error originated from one of the Shopify app test stores, but it has caused our webhook to be deleted... Let's hope they will provide feedback soon and this issue can get some traction.

[juhanaka]

I have the same issue with the orders/fulfilled topic returning 403 seemingly randomly. All other webhook topics work and orders/fulfilled works 99% of the time. It very much seems to be something in how the Shopify node library handles the request or how the requests are sent from Shopify on this particular topic.

[thecodepixi]

@haveneersrobin Hey, please do not tag Shopify employees directly in these threads. These are always on our radar so someone will follow up when/if appropriate. Additionally, I am no longer on the team that manages and contributes our API libraries, so I'm not in a position to assist with this. Take care ✌🏻

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will be closed if no further action occurs in 14 days.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will be closed if no further action occurs in 14 days.

[github-actions[bot]]

We are closing this issue because it has been inactive for a few months. This probably means that it is not reproducible or it has been fixed in a newer version. If it’s an enhancement and hasn’t been taken on since it was submitted, then it seems other issues have taken priority.

If you still encounter this issue with the latest stable version, please reopen using the issue template. You can also contribute directly by submitting a pull request– see the CONTRIBUTING.md file for guidelines

Thank you!