Express webhook not authenticating

[okomarov]

Issue Summary

The validation methods in webhooks won't validate messages initiated by a user on Whatsapp.

Steps to Reproduce

Pretty much an interactive replica of How to secure Twilio webhook URLs in Node.js

1. Go to the sample project on https://replit.com/@OlegAzava/ExpressWithTwilioWebhook#index.js and wake up the Replit
2. Replace the TWILIO_AUTH_TOKEN with a working one
3. Configure a Whatsapp sender in Twilio to send messages to https://ExpressWithTwilioWebhook.olegazava.repl.co/api/message
4. Send a message from within Whatsapp to the target phone number

Code Snippet

const express = require('express')
const twilio = require('twilio')
const app = express()
const port = 3000

app.use(express.urlencoded({ extended: false }));

app.post('/api/message', (req, res, next) => {
  const isValid = twilio.validateExpressRequest(req, process.env['TWILIO_AUTH_TOKEN'])
  console.log(isValid)

  const twilioSignature = req.headers['x-twilio-signature'];
  const params = req.body;
  const url = 'https://ExpressWithTwilioWebhook.olegazava.repl.co'
  const isValid2 = twilio.validateRequest(
    process.env['TWILIO_AUTH_TOKEN'],
    twilioSignature,
    url,
    params
  );
  console.log(isValid2)
  
  if (!isValid && !isValid2) {
    res.status(401).send();
    return next();
  }

  res.set({ 'Content-Type': 'text/plain' });
  res.send('Success');
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Exception/Log

Signatures comparison fails

Technical details:

• twilio-node version: 4.8.0
• node version: v18.12.1

[Hunga1]

I tried to replicate the issue your having, but was only able to find that the twilio.validateExpressRequest method is failing to validate the request. I'm able to get twilio.validateRequest to validate the webhook request successfully.

An issue I see in your code snippet is that your url value is missing the /api/message path. The URL path of the webhook endpoint is required in the URL to validate the request successfully. Updating:

const url = 'https://ExpressWithTwilioWebhook.olegazava.repl.co'

to...

const url = 'https://ExpressWithTwilioWebhook.olegazava.repl.co/api/message'

Should solve your failed request validation with the twilio.validateRequest method.

I'm still investigating why the twilio.validateExpressRequest is failing for us.

[okomarov]

@Hunga1 I tried with both urls and both methods fail. We're looking to migrate off if we can't verify the signature.

To confirm:

• I updated the token to our account's token
• Updated the url
• Pasted the same URL in the Whatsapp sender webhook and texted to the phone number on whatsapp, and both methods fail to verify the signature

Screenshot showing false for both methods (token hidden for obvious reasons): =

Screenshot showing the url of the webhook in Twilio:

[Hunga1]

This issue has been added to our internal backlog to be prioritized. Pull requests and +1s on the issue summary will help it move up the backlog. (Internal ref: DI-2629)

[Hunga1]

@okomarov As a potential workaround, could you try normalizing your webhook URL hostname to lowercase characters (outlined in RFC 3986 - Sec 3.2.2. Host), use that URL in both your webhook URL setting and test app (i.e. https://expresswithtwiliowebhook.olegazava.repl.co/api/message), and call the twilio.validateRequest() method with that normalized URL in your app.

The issue looks like it's being caused by the SDK normalizing the URL hostname to lowercase before generating the signature client-side to compare against the webhook request x-twilio-signature value, which is case-sensitive.

[okomarov]

The issue looks like it's being caused by the SDK normalizing the URL hostname to lowercase before generating the signature client-side to compare against the webhook request x-twilio-signature value, which is case-sensitive.

Hi @Hunga1 thanks for the reply. The test case is an example. We're using in our app all lowercase URLs but I'll try in any case. What other type of normalisations might happen? We do have a '-' like https://hello-world.com/api/webhook

[Hunga1]

The '-' dash character in the hostname should be fine. You'll just want to use lowercase hostnames for both your application and Twilio webhook setting.

[miramo]

Hi, IDK if the problem we're facing is exactly the same as the one mentioned in this issue, but as it seems very similar, I'll try my luck here. If not, let me know and I will make a separate issue.

After upgrading the lib from 3.54.0 to 4.9.0 we ran into the problem that all webhooks failed.

We use the validateRequest method to verify that the request is coming from Twilio.

Where previously (in 3.54.0) validateRequest(authToken, twilioSignature, url, req.body) worked, in 4.9.0 it no longer does, the method returns false.

What does work (in 4.9.0) is validateRequest(authToken, twilioSignature, url, {})

Have we misunderstood something, or does this look like a bug?

For more context, here is the full code we are using:

/**
 * Guard the endpoints supposed to receive only signed requests.
 * For the sake of the example, see signatures made by Twilio:
 * https://twilio.com/docs/usage/webhooks/webhooks-security
 */
@Injectable()
export class TwilioHttpGuard implements CanActivate {
  constructor(private readonly configService: ConfigService) {}

  /**
   * Determine whether the current request has been properly signed, otherwise it must be ignored
   * @param context Nest context; allow getting the HTTP request
   * @throws {ForbiddenException} When the request has not valid signature
   * @return `true` if the request is signed
   */
  canActivate(context: ExecutionContext) {
    const req: IGojobRequest = context.switchToHttp().getRequest();

    if (!this.isHttpRequestVerified(req)) {
      throw new ForbiddenException('[TWILIO] Request must be signed');
    }
    return true;
  }

  private isHttpRequestVerified(req: Request): boolean {
    const authToken = this.configService.get('TWILIO_AUTH_TOKEN');

    const url = `${req.get('x-forwarded-proto')}://${req.get('host')}${req.url}`;
    const signatureHeader = req.get('x-twilio-signature') ?? '';

    return validateRequest(authToken, signatureHeader, url, req.body);
  }
}

[yoann-gendrey]

I haven't been verified the webhook signature before (with previous version) as I'm newly integrating Twilio webhook, but I did pretty much the exact same thing as @miramo and also facing same issue, the request is not validated as expected.

I also tried with the validateRequestWithBody with no success:

    return validateRequestWithBody(
      process.env.TWILIO_AUTH_TOKEN,
      request.headers['x-twilio-signature'] as string,
      this.statusCallback,
      request.rawBody.toString(),
    );

[yoann-gendrey]

I dug a little bit more on that and find a way to validate the signature. Honestly I don't know what are the motivation from that function which might be the bug ? https://github.com/twilio/twilio-node/blob/d037d6f445b210cdc455a01dee3d61a664714a20/src/webhooks/webhooks.ts#L230

It does fit with what @miramo said, but passing empty object as param seems wrong to validateRequest. Anyway in all cases both validateRequest OR validateRequestWithBody were not working as the empty params were passed, and it also makes sense that validateRequestWithBody as both validateRequest and validateBody should be valid.

Instead I used https://github.com/twilio/twilio-node/blob/d037d6f445b210cdc455a01dee3d61a664714a20/src/webhooks/webhooks.ts#L135

That one compute the sig exactly how it's done on Twillio's side, then you'll end up with a validation function that should looks like that:

    const signature = getExpectedTwilioSignature(
      process.env.TWILIO_AUTH_TOKEN,
      this.statusCallback,
      request.body,
    );
    return signature === request.headers['x-twilio-signature'];

Pay also attention, don't know which framework you are using, but based on what you are using you might used sometimes request.rawBody or request.body. What you should know is that the getExpectedTwilioSignature is expecting the payload received from the webhook as a javascript object. The 3 parameters are respectively token, webhook_url, and body (parsed js object)

Then function is returning the proper signature and just compare it to the one you are getting from the header.

[okomarov]

@AsabuHere given this seems a regression as pointed out by https://github.com/twilio/twilio-node/issues/924#issuecomment-1487393481 and that the verification flow has multiple branches which are not clearly explained in the code, I reckon there should be at least some guidance from the twilio team on how to go about it (if PR, who's going to review it and when) or a communication of what priority this issue has internally so we can at least fork and patch.

[miramo]

Since my last comment, we found something interesting in the security documentation.

What we noticed is that depending on "where" the webhook is called from (studio flow, function or callback configured for a number) the behaviour is not the same.

So, according to what we noticed, we implemented two different guards depending on the webhook we were trying to validate.

For the webhook called in a Studio Flow (Content-Type is application-json) :

  private isHttpRequestVerified(req: RawBodyRequest<Request>): boolean {
    const authToken = this.configService.get('TWILIO_AUTH_TOKEN');

    const url = `${req.get('x-forwarded-proto')}://${req.get('host')}${req.url}`;
    const signatureHeader = req.get('x-twilio-signature');

    const rawBody = req.rawBody?.toString('utf8') ?? '';
    return validateRequestWithBody(authToken, signatureHeader, url, rawBody);
  }

And for the callback that is configured for a number:

  private isHttpRequestVerified(req: Request): boolean {
    const authToken = this.configService.get('TWILIO_AUTH_TOKEN');

    const url = `${req.get('x-forwarded-proto')}://${req.get('host')}${req.url}`;
    const signatureHeader = req.get('x-twilio-signature');

    return validateRequest(authToken, signatureHeader, url, req.body);
  }

IDK if it's the right solution and that's how we had to use these methods, but that's what worked for us.

[okomarov]

Going back to my reproducible test, we seem to manage to validate with validateRequest but not with validateExpressRequest. Two findings:

• the TWILIO_AUTH_TOKEN should be from the main dashboard (clicking top-left > go to main dashboard) NOT from the keys and credentials (screenshot below) This was just such a huge waste of time. Their dedicated Tokens also don't work as expected... I've no idea why there are 3-4 different tokens but only one works

• the difference in casing of the URL does NOT matter from our tests

[andrewkouri]

I am also having this issue (NextJS 13) but I am not seeing a difference between the two Auth Tokens?

[trentonsnyder]

I'm seeing

validateRequest(TWILIO_AUTH_TOKEN, sig, fullURL.href, {})

is true for a GET but false for a POST. The 4th argument says @param params — the parameters sent with this request But there are no parameters on the POST. So I'd think it should be working the opposite 🤷

using 4.19

[patrykmaron]

We are also using validateRequest. We get an POST when WhatsApp message comes in, it works perfectly fine when whatsapp message comes through, until it's an "reply" to an message in a chat and the validation fails