Troubleshooting GitHub Webhooks

Introduction

In a previous article, I demonstrated the use of GitHub webhooks in a simple tutorial using a sample API that logged webhook events. In this article, we will learn how to debug common issues with GitHub webhooks using the same sample project. This time, I have introduced some bugs into the project that we are going to learn how to debug.

Troubleshooting GitHub webhooks: Requirements checklist

To begin there are a few things you need to have or set up, including:

• A GitHub repository
• Node.js installed on your system to run the sample project
• A publicly accessible URL to the API endpoint. We will use the Hookdeck CLI to achieve this. To install the Hookdeck CLI, check out this page.
• Clear visualization of error messages. The Hookdeck CLI will help us with event pages where error details can be inspected.
• A text editor for editing code

With this setup, you have an environment in which you can conveniently troubleshoot your GitHub webhooks.

Troubleshooting GitHub webhooks with Hookdeck

Cloning and running a demo API

The sample API we will use is available on the Hookdeck GitHub repository. Follow the instructions to set it up.

Clone the repository by running the following command:

git clone --single-branch --branch github-webhooks-debugging https://github.com/hookdeck/nodejs-webhook-server-example.git

Navigate to the root of the project and install the required dependencies by running the following commands:

cd nodejs-webhook-server-example
npm install

When the installation completes, run the Node.js server with the following command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project:

• /log-github-webhook: This is the endpoint that will be receiving the GitHub webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
• /fetch-webhooks-logs: This endpoint can be called to retrieve a collection of the logged webhook data.

Getting a GitHub webhook URL

The next step is to use the CLI to generate a webhook URL that points to the running API application. To do this, run the following command:

hookdeck listen 1337

This command starts an interactive session where the CLI collects information about the endpoint you're about to create. Below are the questions and the answers you should supply to each question. Ensure to hit the Enter key after each answer.

• What source should you select? Ans: select Create new source
• What should your new source label be? Ans: type the text GitHub
• What path should the webhooks be forwarded to (i.e.: /webhooks)? Ans: type /log-github-webhook
• What's the connection label (i.e.: My API)? Ans: type My Github Response Server

With this information, the CLI will begin the process of generating the URL and once it's done, you will see the URL printed to the screen and the CLI indicating that it is ready to receive requests.

Note: You will need to use the guest Login URL link in the console to access the dashboard. Copy and paste this into your browser to begin a guest login session.

Confirming webhook delivery

To begin receiving webhooks, you need to set up a webhook on your GitHub repository.

Go to Settings → Webhooks. On the Webhooks page, click on the Add webhook button on the top right-hand corner.

On the webhook form displayed, paste the webhook URL generated by the Hookdeck CLI into the Payload URL field.

Fill the remaining fields as follows:

• Content type: Select application/json so that you can receive the payload as a JSON object.
• Secret: You can leave this blank, but for this tutorial enter 1234ABCD.
• SSL Verification: Leave this as the default option, Enable SSL verification.
• Which events would you like to trigger this webhook: This is the point where you subscribe for a GitHub event on your repository. For this tutorial, select the Just the push event option as we are only subscribing to the push event. You can either subscribe for all events or a subset of the events using the other two options.
• Active: Leave this checked to receive event details when the GitHub webhook is triggered.

See the complete webhook form below:

Click the Add webhook button to complete the process.

With this setup, anytime you push code to your repository, a webhook request will be fired to the specified webhook URL.

Immediately after you complete registration for a webhook in the step above, a ping webhook request will be fired to your webhook URL. Because we are using the Hookdeck CLI, we would already see the ping webhook request logged on the terminal where the Hookdeck CLI is running, as shown below:

This confirms that we are successfully receiving our webhook request. However, did you notice the status code indicated on the webhook request? Yeah, a 404.

Let's take a look at how to deal with this error in the next section.

Troubleshooting GitHub webhook "not found" error

Our first webhook attempt resulted in a 404 error on the destination endpoint. This tells us that even though we are successfully receiving our webhooks, it cannot locate the endpoint specified, or the endpoint does not exist. 404 errors can be fixed, most of the time, by checking for typos or misspellings in the specified route name. Worst case scenario, the route truly does not exist and needs to be created.

Use the event page link displayed on the CLI to go to the event page, where you will see a page similar to the one below:

Scroll down to the red status code badge under the Attempts section and click on it to reveal details about the error on the right-hand side of the screen, as shown below:

This helps us see the actual response from the running server, and it also confirms that the specified endpoint cannot be found.

The endpoint specified for the webhook to hit is /log-github-webhook, which can be found in the routes.js file at the root of the project directory.

router.post("/log-github-webhoo", async function(req, res) {
  //console.log(req.body);

  const payload = req.body;

  let webhook_info = {
    repo : payload.repository.name,
    author : payload.sender.login,
    time : payload.head_commit.timestamp
  }

  const save_webhook = await req.db
  .collection("webhooks")
  .insertOne(webhook_info);

  res.status(201).send({
    message: "Webhook Event successfully logged"
  });
});

Taking a close look at the route handler, you will notice that there is a typo just at the end of the endpoint name: a missing 'k'. Simply add the missing character and save the file.

You will need to restart the Node.js server to allow the changes to take effect. Shut down the server using Ctrl + C and restart it with the npm start command.

Now, fire a new webhook to your endpoint by pushing a commit to your GitHub repository. Once this is done, check your Hookdeck CLI session for the new webhook received. You will see an entry similar to the one below:

As seen from the above webhook received, we have been able to clear the 404 error. However, we now have a new error: a 401 error. Let's deal with this in the following section.

Troubleshooting GitHub webhook error 401

HTTP 401 errors indicate that an unauthorized action is being performed. Before making any assumptions, let's inspect the server response using the event's page. Copy the event page link from the Hookdeck CLI session and load it in your browser, and you will see a screen similar to the one below:

Scroll down to the red status code badge under the Attempts section and then click on it. You will see the server response details on the right-hand side as shown below:

Reading the response message, we can see that we are failing the security check on the webhook payload. Because we defined an API secret, GitHub sends the X-Hub-Signature-256 header. This header contains an encrypted version of the real payload and has to be validated against the unencrypted payload received by using the secret key.

This is a security check to prevent attackers from replacing the actual payload with a malicious one with the intention of corrupting our API.

Because we know that we are receiving the actual payload from GitHub since we set everything up and triggered the push event, we know it's not an attacker's payload causing the 401 error. Something else must be wrong with our setup.

Let's take a look at the validation logic on our API. This can be found in the server.js file, as shown below:

//Validate payload
function validatePayload(req, res, next) {

    if(req.method == "POST"){
        if (!req.rawBody) {
            return res.status(400).send({
                message: "Request Body empty"
            });
        }

        const sig = Buffer.from(req.get(sigHeaderName) || '', 'utf8')
        const hmac = crypto.createHmac(sigHashAlg, secret)
        const digest = Buffer.from(sigHashAlg + '=' + hmac.update(req.rawBody).digest('hex'), 'utf8');

        if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
            return res.status(401).send({
                message: `Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
            });

        }
    }

    return next()

}
app.use(validatePayload);

There are a couple of variables being used here in our code that are worth checking out to confirm that they are referencing the appropriate values. These are:

• sigHeaderName: This represents the signature header name sent by GitHub.
• sigHashAlg: This is the algorithm used for the encryption.
• secret: This is the API secret set on our GitHub webhook.

These values are set on lines 10-12 in the server.js file. Lets take a look at these definitions:

const sigHeaderName = 'X-Hub-Signature-256';
const sigHashAlg = 'sha256';
const secret = "XXXXXX";

After observing these values, have you been able to catch the error? If the secret is not the same as the one we set in our webhook form, the validation will always fail.

GitHub does not allow you to view the API secret you set in your webhooks a second time — you can only reset it. This is why you need to make sure that you remember the value or store it somewhere secure.

For this practice, we know we set the secret as a simple 1234ABCD string. In real-world applications, you want to set a more complicated secret and reference it from an environment variable in your code.

Change the value of the secret to the right value, save the file, and restart the server. Now trigger another webhook by pushing a new commit to your GitHub repository. You will see a webhook entry similar to the one below on your Hookdeck CLI session:

Finally, we have a successful operation on our API. Load the event page, which should display a page similar to the one below:

Now click on the green status badge below the Attempts section to show the server response. The server returns a successful message, as shown below:

This time, when you visit the endpoint /fetch-webhooks-logs, you will see a collection like the one below:

Troubleshooting GitHub webhook "invalid HTTP response 400" error

Another error that can occur with GitHub webhooks is a 400 HTTP error, which indicates a bad request. One thing to note when working with GitHub webhooks is that GitHub is your client. Your application, or whichever system (SaaS apps, CI/CD server, etc.) that you're integrating GitHub webhooks with, is the server.

Thus, when you see a bad request, it means that your server is not receiving what it expects from GitHub.

The first thing you want to check is the format in which you're sending your webhook payload. GitHub allows you to set the Content-type to either application/json or application/x-www-form-urlencoded .

Make sure that the value you configured in your webhook form matches the content type your server is expecting.

Sometimes, 400 errors arise from empty payloads being sent by GitHub. If your server expects a payload to be sent, make sure that the Active option on your GitHub webhook form is checked.

Conclusion

Just like any other architectural component of an application, webhooks run into errors. Debugging is considered part of the workflow when developing applications or setting up application infrastructure. As errors are inevitable, knowing what to look out for when a certain error occurs and having the right tools to debug the situation eases the pains experienced in troubleshooting.

You also save time and push out solutions faster!