Mandantory GDPR webhooks

[cmnstmntmn]

There is no easy way to find the topic for a mandatory web-hook;

From documentations the list of these 3 mandantory webhooks is:

list 1:

• customers/data_request: Requests to view stored customer data
• customers/redact: Requests deletion of customer data
• shop/redact: Requests deletion of shop data

however, all examples includes APP_UNINSTALLED but there is no example with the other three.

My question so far.. Is there a correlation between list 1 and this list https://shopify.dev/api/admin-graphql/2021-10/enums/webhooksubscriptiontopic ?

[arfanliaqat]

@cmnstmntmn, This is definitely confusing at best and took me quite sometime to figure out as well. Here is how to do it

1. Goto your shopify partners
2. Open the app where you want to add these hooks
3. Goto App Setup
4. Scroll all the way down to following section

5. And here you can add ur endpoints for handling these mandatory hooks

Hope this helps

[bishpls]

Adding onto this issue: something else missing from the documentation is guidance on verifying that the mandatory GDPR webhooks come from Shopify.

The official docs include examples in Ruby, PHP, and Python here: https://shopify.dev/apps/webhooks/configuration/https#verify-the-webhook

but since these webhooks don't go through Shopify.Webhooks.Registry.process (which is handling the hmac validation to confirm the requests are, in fact, coming from Shopify), it would be very useful to have an official example of manually going through the hmac validation process to verify the webhooks! (Or, even better, an invokable util function we can use to do so)

If I'm understanding the process correctly, validating general requests / oAuth functions a little differently from validating webhooks, which is why Registry.process doesn't use hmac-validator.ts; is it best practice at present to essentially repackage the relevant code from Registry.process for mandatory GDPR webhooks?

[bishpls]

I believe the following should be a minimally-viable example?

api.use(express.json()); // Clashes with Shopify.Webhooks.Registry.process, so this MUST be AFTER default /webhooks route
api.post('/customers/data', async(req, res) => {
  const generatedHash = crypto.createHmac('SHA256', process.env.SHOPIFY_API_SECRET).update(JSON.stringify(req.body), 'utf8').digest('base64');
  // this should reference ShopifyHeader.Hmac exported type for future-proofing
  const hmac = req.get('X-Shopify-Hmac-Sha256');
  const safeCompareResult = Shopify.Utils.safeCompare(generatedHash, hmac);
  console.log(safeCompareResult); // Should be true for valid requests

  res.status(200).send();
  // Handle business logic of GDPR endpoint after sending status 200 in live application
});

[TheSecurityDev]

@bishpls Dude you're a life saver. I literally needed to know this exact thing, since handling the mandatory webhooks seems different than using the normal Shopify.Webhooks.Registry.process() method, and I find this thread with your reply just 3 hours old!

I think something as important and common as this should be handled by the library in some way. I don't know why we can't just use the normal webhook handler functions for this :/

And yes, this whole thing needs much more documentation, and there needs to be a way to test the webhooks functionality beforehand since every app needs to handle these webhooks.

[bishpls]

You can test the customers/data endpoint quite easily!

Assuming your app is installed on a development store you have access to, navigate to Customers > click any Customer > check the right panel for the "Customer Privacy" block; 'View customer data' will immediately send a payload to the customers/data endpoint set up for your installed app(s). "Erase Personal Data" hits the customers/redact endpoint, but is subject to the delay of ten-days-or-six-months, so it's not particularly useful for testing.

[bishpls]

HOWEVER it's worth calling attention (yet again) to the fact that body-parser / express.json breaks the standard Shopify.Webhooks.Registry.process function, so if you're using either to parse the payload from the GDPR webhooks, you'll either want to run app.use(express.json()) AFTER /webhooks but before the GDPR routes, or just inline as a substack call for the GDPR routes.

I've set mine up via the latter, like so:

api.post('/customers/redact', express.json(), verifyWebhookRequest, async(req, res) => {
  res.status(200).send();
  // Business logic goes here
}

EDIT:

And for reference, here's my verifyWebhookRequest function:

RE-EDIT: Updating to use Shopify.Context.API_SECRET_KEY

function verifyWebhookRequest(req, res, next) {
  try {
    const generatedHash = crypto.createHmac('SHA256', Shopify.Context.API_SECRET_KEY).update(JSON.stringify(req.body), 'utf8').digest('base64');
    const hmac = req.get(ShopifyHeader.Hmac); // Equal to 'X-Shopify-Hmac-Sha256' at time of coding

    const safeCompareResult = Shopify.Utils.safeCompare(generatedHash, hmac);

    if (!!safeCompareResult) {
      console.log('hmac verified for webhook route, proceeding');
      next();
    } else {
      console.log('Shopify hmac verification for webhook failed, aborting');
      return res.status(401).json({ succeeded: false, message: 'Not Authorized' }).send();
    }   
  } catch(error) {
    console.log(error);
    return res.status(401).json({ succeeded: false, message: 'Error caught' }).send();
  }
}

[TheSecurityDev]

@bishpls Just a suggestion, I would change process.env.SHOPIFY_API_SECRET_KEY to Shopify.Context.API_SECRET_KEY, so it always works, no matter the env variable name.

[ZPetrovich]

@bishpls I can not test View Customer Data by clicking button you suggested. My endpoint just not invoked by Shopify at all! :(

[stephenkeable]

Might be of some help to people using the next.js / koa.js solutions, I ended up using the following approach in the server/server.js

Tweaked version of @bishpls function

function verifyWebhookRequest(body) {
  try {
    const generatedHash = crypto
      .createHmac("SHA256", Shopify.Context.API_SECRET_KEY)
      .update(JSON.stringify(body), "utf8")
      .digest("base64");
    const hmac = req.get(ShopifyHeader.Hmac); // Equal to 'X-Shopify-Hmac-Sha256' at time of coding
    const safeCompareResult = Shopify.Utils.safeCompare(generatedHash, hmac);
    if (!!safeCompareResult) {
      return true;
    } else {
      return false;
    }
  } catch (error) {
    return false;
  }
}

Then using the koa-body package to add ctx.request.body.

Then add new routes for each of the GDPR web hooks after the existing /webhooks one:-

router.post("/webhook-gdpr-request", koaBody(), (ctx) => {
    if (verifyWebhookRequest(ctx.request.body) === true) {
      ctx.res.statusCode = 200;
// do something with the ctx.request.body
    } else {
      ctx.res.statusCode = 401;
    }
  });

[akeans-mgs]

Our app was rejected by Shopify's automated submission test immediately after submission with the same reason of

App must verify the authenticity of the request from Shopify. Expected HTTP 401 (Unauthorized), but got HTTP 405 from https://8235cf20c428.ngrok.io/webhook/gdpr/shop_redact. Your app's HTTPS webhook endpoints must validate the HMAC digest of each request, and return an HTTP 401 (Unauthorized) response when rejecting a request that has an invalid digest.

Error I received

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined in
at Hmac.update (internal/crypto/hash.js:84:11)
at receiveWebhookMiddleware( \node_modules@shopify\koa-shopify-webhooks\build\cjs\receive.js:32:63 )
at dispatch( \node_modules@shopify\koa-shopify-webhooks\node_modules\koa-compose\index.js:42:32 )
at bodyParser \node_modules\koa-bodyparser\index.js:95:11)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at \node_modules\koa-mount\index.js:58:5

You can check the code here. https://github.com/akeans-mgs/mgs_testing

refer the line https://github.com/akeans-mgs/mgs_testing/blob/main/server/server.js#L85

[smotched]

hey @stephenkeable it doesn't seem like the function defines neither req nor ShopifyHeader

[stephenkeable]

hey @stephenkeable it doesn't seem like the function defines neither req nor ShopifyHeader

Yeh it assumes some globals from the server/server.js file. You could change the signature to pass these in instead, so: function verifyWebhookRequest(body, req) { ... } Or something similar

[akeans-mgs]

Hi guys, @stephenkeable @smotched kindly help me with the above issue - i have mentioned.

[stephenkeable]

@akeans-mgs sorry in my solution I haven't used koa-shopify-webhooks you might find some help for that in the https://github.com/Shopify/quilt issues

[akeans-mgs]

@stephenkeable If possible share some code to integrate Shopify GDPR webhook in Node + React.

[swherden]

@akeans-mgs you can use receiveWebhook method from @shopify/koa-shopify-webhooks. Example code from my Koa server - maybe it is of value for you:

import { receiveWebhook } from '@shopify/koa-shopify-webhooks';
...
const webhook = receiveWebhook({ secret: SHOPIFY_API_SECRET });
router.post(
        <enter your webhook path e.g /webhooks/gdpr/customerdatarequest>,
        webhook,
        (ctx: any) => {
            console.log(ctx.request);
            const { shop_domain, topic, payload } = getWebhookConfig(ctx);
            console.log(shop_domain, topic, payload);
           //do important GDPR things here
        }
    );
...
function getWebhookConfig(ctx: any) {
    const shop_domain = ctx.state.webhook.domain;
    const topic = ctx.state.webhook.topic;
    const payload = ctx.state.webhook.payload;
    return { shop_domain, topic, payload };
}

[daviareias]

Thanks guys, the following code worked for me (you can erase the console.logs later, I've added it just to understand how it works):

function verifyWebhookRequest(body,req) {
    try {
      const generatedHash = crypto
        .createHmac("SHA256", Shopify.Context.API_SECRET_KEY)
        .update(JSON.stringify(body), "utf8")
        .digest("base64");
      const ShopifyHeader = 'x-shopify-hmac-sha256';
      const hmac = req.get(ShopifyHeader); 
      const safeCompareResult = Shopify.Utils.safeCompare(generatedHash, hmac);
      if (!!safeCompareResult) {
        console.log('Safe')
        return true;
      } else {
        console.log('Not Safe')
        return false;
      }
    } catch (error) {
      console.log('error', error)
      return false;
    }
 }

router.post("/customers/data_request", koaBody(), (ctx) => {
  if (verifyWebhookRequest(ctx.request.body,ctx.request) === true) {
    console.log('verified :)')
    ctx.res.statusCode = 200;
// do something with the ctx.request.body
  } else {
    console.log('Not verified')
    ctx.res.statusCode = 401;
  }
});

[akeans-mgs]

@akeans-mgs you can use receiveWebhook method from @shopify/koa-shopify-webhooks. Example code from my Koa server - maybe it is of value for you:

import { receiveWebhook } from '@shopify/koa-shopify-webhooks';
...
const webhook = receiveWebhook({ secret: SHOPIFY_API_SECRET });
router.post(
        <enter your webhook path e.g /webhooks/gdpr/customerdatarequest>,
        webhook,
        (ctx: any) => {
            console.log(ctx.request);
            const { shop_domain, topic, payload } = getWebhookConfig(ctx);
            console.log(shop_domain, topic, payload);
           //do important GDPR things here
        }
    );
...
function getWebhookConfig(ctx: any) {
    const shop_domain = ctx.state.webhook.domain;
    const topic = ctx.state.webhook.topic;
    const payload = ctx.state.webhook.payload;
    return { shop_domain, topic, payload };
}

@swherden if possible, kindly share the server.js file to have close look at it.

[akeans-mgs]

@swherden Thanks for replying. Kindly accept my request.

I have invited you to my sample shopify app repo , kindly accept it. And we can do the needful to work with GDPR Webhook Integration.

[yashsony]

Just one add on to this conversation You don't create webhook subscriptions to mandatory webhooks. Instead, you configure mandatory webhooks in your Partner Dashboard as part of your app setup. You just need to define endpoints of mandatory webhooks.

[ismailalabou]

Thanks guys, the following code worked for me (you can erase the console.logs later, I've added it just to understand how it works):

function verifyWebhookRequest(body,req) {
    try {
      const generatedHash = crypto
        .createHmac("SHA256", Shopify.Context.API_SECRET_KEY)
        .update(JSON.stringify(body), "utf8")
        .digest("base64");
      const ShopifyHeader = 'x-shopify-hmac-sha256';
      const hmac = req.get(ShopifyHeader); 
      const safeCompareResult = Shopify.Utils.safeCompare(generatedHash, hmac);
      if (!!safeCompareResult) {
        console.log('Safe')
        return true;
      } else {
        console.log('Not Safe')
        return false;
      }
    } catch (error) {
      console.log('error', error)
      return false;
    }
 }

router.post("/customers/data_request", koaBody(), (ctx) => {
  if (verifyWebhookRequest(ctx.request.body,ctx.request) === true) {
    console.log('verified :)')
    ctx.res.statusCode = 200;
// do something with the ctx.request.body
  } else {
    console.log('Not verified')
    ctx.res.statusCode = 401;
  }
});

if you don't mind me asking, is there a reason behind making ShopifyHeader static ??

[Ankita-K-Chauhan]

My response is not working for mandatory webhook any suggestion. I create the routes for the response method but when I hit the "customers_data_request" my controller is not called but job is called
I check the routes it looks fine

[justinhenricks]

Adding onto this issue: something else missing from the documentation is guidance on verifying that the mandatory GDPR webhooks come from Shopify.

The official docs include examples in Ruby, PHP, and Python here: https://shopify.dev/apps/webhooks/configuration/https#verify-the-webhook

but since these webhooks don't go through Shopify.Webhooks.Registry.process (which is handling the hmac validation to confirm the requests are, in fact, coming from Shopify), it would be very useful to have an official example of manually going through the hmac validation process to verify the webhooks! (Or, even better, an invokable util function we can use to do so)

If I'm understanding the process correctly, validating general requests / oAuth functions a little differently from validating webhooks, which is why Registry.process doesn't use hmac-validator.ts; is it best practice at present to essentially repackage the relevant code from Registry.process for mandatory GDPR webhooks?

so glad I found this thread.. I have been so confused all day because of lack of documentation on these GDPR webhooks.. thank you !!!!

[i-moreno]

My 2 cents to this issue:

1. Add endpoint routes in your app dashboard as @arfanliaqat mentions
2. I create a file to handle gdpr webhooks routes
3. Add your middleware to verify the HMAC
4. Update your server file, make sure to call express json before grpr routes.

gdpr webhooks routes

import express from "express";
import {
  customerDataRequest,
  customerRedact,
  shopRedact,
} from "../webhooks/gdpr.js";

const gdprRoutes = express.Router();

gdprRoutes.post("/:topic", async (req, res) => {
  const { body } = req;
  const { topic } = req.params;
  const shop = req.body.shop_domain;

  let response;

  switch (topic) {
    case "customers_data_request":
      response = await customerDataRequest(topic, shop, body);
      break;
    case "customers_redact":
      response = await customerRedact(topic, shop, body);
      break;
    case "shop_redact":
      response = await shopRedact(topic, shop, body);
      break;
    default:
      console.log("--> Unidentified GDPR Topic");
      break;
  }

  if (response && response.success) {
    res.status(200).send();
  } else {
    res.status(400).send("Error with mandatory GDPR webhooks");
  }
});

export default gdprRoutes;

HMAC middleware verification

import crypto from "crypto";
import { Shopify } from "@shopify/shopify-api";

export const hmacVerify = (req, res, next) => {
  try {
    const generateHash = crypto
      .createHmac("SHA256", process.env.SHOPIFY_API_SECRET)
      .update(JSON.stringify(req.body), "utf8")
      .digest("base64");

    const hmac = req.headers["x-shopify-hmac-sha256"];

    if (Shopify.Utils.safeCompare(generateHash, hmac)) {
      console.log("HMAC successfully verified for webhook route.");
      next();
    } else {
      console.log("Shopify hmac verification for webhook failed, aborting.");
      return res.status(401).send();
    }
  } catch (error) {
    console.log("--> HMAC ERROR", error);
  }
};

Server extract

  app.use(express.json());
  app.use("/gdpr", hmacVerify, gdprRoutes);

[mkevinosullivan]

Using the latest CLI to generate a Node app template will include generic GDPR handlers and registration:

setup method https://github.com/Shopify/shopify-app-template-node/blob/cli_three/web/gdpr.js

index.js extract:

// This sets up the mandatory GDPR webhooks. You’ll need to fill in the endpoint
// in the “GDPR mandatory webhooks” section in the “App setup” tab, and customize
// the code when you store customer data.
//
// More details can be found on shopify.dev:
// https://shopify.dev/apps/webhooks/configuration/mandatory-webhooks
setupGDPRWebHooks("/api/webhooks");

https://github.com/Shopify/shopify-app-template-node/blob/cli_three/web/index.js#L56-L62

[stephenkeable]

@mkevinosullivan the latest CLI is much better for explaining the GDPR web hooks, thanks!

[olivierbouchard]

Using the latest CLI to generate a Node app template will include generic GDPR handlers and registration:

setup method https://github.com/Shopify/shopify-app-template-node/blob/cli_three/web/gdpr.js

index.js extract:

// This sets up the mandatory GDPR webhooks. You’ll need to fill in the endpoint
// in the “GDPR mandatory webhooks” section in the “App setup” tab, and customize
// the code when you store customer data.
//
// More details can be found on shopify.dev:
// https://shopify.dev/apps/webhooks/configuration/mandatory-webhooks
setupGDPRWebHooks("/api/webhooks");

https://github.com/Shopify/shopify-app-template-node/blob/cli_three/web/index.js#L56-L62

@mkevinosullivan How do you configure your endpoint in the app (if you have to) and what are the URLs you write in the app setup?

[mkevinosullivan]

@olivierbouchard To set up the /api/webhooks endpoint: https://github.com/Shopify/shopify-app-template-node/blob/cli_three/web/index.js#L79-L93

  // Do not call app.use(express.json()) before processing webhooks with
  // Shopify.Webhooks.Registry.process().
  // See https://github.com/Shopify/shopify-api-node/blob/main/docs/usage/webhooks.md#note-regarding-use-of-body-parsers
  // for more details.
  app.post("/api/webhooks", async (req, res) => {
    try {
      await Shopify.Webhooks.Registry.process(req, res);
      console.log(`Webhook processed, returned status code 200`);
    } catch (e) {
      console.log(`Failed to process webhook: ${e.message}`);
      if (!res.headersSent) {
        res.status(500).send(e.message);
      }
    }
  });

what are the URLs you write in the app setup?

I'm not sure what you mean by this.

[olivierbouchard]

@mkevinosullivan I have this in my code but with this, what are the endpoints you write in the app setup? Sorry btw, my base langage is french.

[mkevinosullivan]

Oh, do you mean the App Setup in your Partner Dashboard?

For these, it should be your App URL (same as configured at the top of the App Setup page), followed by /api/webhooks, e.g., https://<app host name>/api/webhooks