X-Hub-Signature is not getting verified

[anshulpatel25]

Description

The webhook lambda shows 'Unable to verify signature!' in the logs. The X-Hub-Signature which is generated by Github enterprise is not getting verified in the webhook lambda.

I have already verified the secret value is same in webhook configuration in Github enterprise and AWS SSM

Software version

• terraform-aws-github-runner: v1.5.0
• Github Enterprise version: 3.5.1

[anshulpatel25]

Upon further investigation, following changes worked for my setup

diff --git a/modules/webhook/lambdas/webhook/src/webhook/handler.ts b/modules/webhook/lambdas/webhook/src/webhook/handler.ts
index 983a11c..72d6cd4 100644
--- a/modules/webhook/lambdas/webhook/src/webhook/handler.ts
+++ b/modules/webhook/lambdas/webhook/src/webhook/handler.ts
@@ -96,7 +96,7 @@ async function verifySignature(
   body: string,
   environment: string,
 ): Promise<number> {
-  const signature = headers['x-hub-signature'] as string;
+  const signature = headers['x-hub-signature-256'] as string;
   if (!signature) {
     logger.error(
       "Github event doesn't have signature. This webhook requires a secret to be configured.",
@@ -110,7 +110,7 @@ async function verifySignature(
   const webhooks = new Webhooks({
     secret: secret,
   });
-  if (!(await webhooks.verify(body, signature))) {
+  if (!(await webhooks.verify(JSON.parse(body), signature))) {
     logger.error('Unable to verify signature!', LogFields.print());
     return 401;
   }

[npalm]

Quickly checked the events sent from github.com, they contain both a sha1 and a sha255

X-Hub-Signature: sha1=
X-Hub-Signature-256: sha256=

Need to check, but indeed maybe we should upgrade the webhook to use the sha256 as default (maybe with fall back to old one). We don't sue GHES, so can't test the module on GHES.

[anshulpatel25]

Thanks @npalm for providing insights, Octokit itself says that sha1 is supported for legacy reasons reference: https://github.com/octokit/webhooks-methods.js/#sign

[michaelstepner]

Hi @npalm, my team is currently upgrading from a very old version of philips-labs/terraform-aws-github-runner to the latest (v1.7.0) and we're encountering this issue as well. I've described our software versions below.

The effect of this bug is that the "scale up" lambda never gets successfully triggered. So runners aren't starting at all. This seems like a major bug if it's affecting other users too.

We've also confirmed that the value of the webhook secret is the same in the Github App configuration and in the terraform state. Unlike @anshulpatel25, we are using Github.com not Github Enterprise.

Do you have an update on the bugfix above or any workarounds? In the meantime, our plan is to identify which older version of the terraform-aws-github-runner module predates the introduction of this bug, and upgrade to that older version.

Software version

terraform-aws-github-runner module: v1.7.0 Terraform version: 1.2.4 Terraform module hashicorp/aws ~> 4.15 Github.com hosted repo

[michaelstepner]

With further debugging, it seems that the issue we're encountering is not the same as @anshulpatel25: the patch proposed above didn't resolve the issue. We're debugging further.

[bjonnh-work]

@anshulpatel25 could you make a PR out of your changes?

[anshulpatel25]

Sure @bjonnh-work , I will do this over the weekend

[anshulpatel25]

Hey @npalm, can you please help with the review: https://github.com/philips-labs/terraform-aws-github-runner/pull/2434? CC: @bjonnh-work

[bjonnh-work]

Thank you @anshulpatel25!