How to add Webhooks to any API

While working on part 3 for my Notion + Qovery series, I was faced with an issue. How could I get notified when a Qovery application status changes, and how to know if a Notion database was updated?

The common solution to this problem is to provide a Webhook. According to the Wikipedia definition:

A webhook in web development is a method of augmenting or altering the behavior of a web page or web application with custom callbacks.

This means, a system will proactively call some endpoint to notify another application whenever some data changed, or some event happened. The receiver application can then act on it.

But neither Qovery nor Notion provide webhooks at the moment, and that’s the case for many other products. Since this is a recurring need for me I decided to create a simple tool, instead of re-inventing the wheel for every project that needs it.

Introducing CaptainHook

The tool I created is called CaptainHook. It’s a simple NodeJS service that will poll API endpoints and forward the response to an arbitrary URL whenever it changes. It’s based on the concept of Trackers. You can create any number of trackers, each of them being responsible to monitor an endpoint. You can check the code on GitHub. It’s very easy to use and the README provides all the information needed. Feel free to use it for your own purpose, but keep in mind that I created it for my personal needs, and not for production.

My goal for this article is not to detail how to use CaptainHook, but to explain the underlying principles so you can implement your own.

How does it work?

CaptainHook is not rocket science and only does a few things:

• Poll an API endpoint at a configurable interval.
• Compare the response from the endpoint to the previous one.
• If the response changed, post the body to an URL and store it.

Polling

Unfortunately, there is no dark magic allowing us to create webhooks out of thin electrons. So to get data changes, we need to act like an annoying project manager poking your shoulder to know if you finished working on your feature all day long (Pro-Tip: If you can relate, find another job!).

But in our case, it’s our only choice so let’s be toxic. We will send requests to the API endpoint we want to monitor on a regular basis. The interval we poll at depends on many parameters:

• How often do I need to get data updates?
• Are there rate limits on the target API?
• Is it worth using bandwidth for my case?

Ideally, you should avoid polling too often. Creating load on the target API server for no reason is always a bad idea and a waste of resources. Even if there is no documented rate-limiting in place, sending requests every 10ms could identify you as a DoS attacker and get you blocked. Also, it’s not very kind. Let’s be responsible. So how often do you really need those updates? Chose carefully.

Possible implementation in NodeJS could be as simple as a setInterval (that’s how CaptainHook works). Other languages could use a loop with a sleep. Or you could go crazy and use worker queues, Erlang gen_server … This part really depends on your scaling needs and the technology you’re using.

Compare

Once you get a response from your API, the next step is to check if the payload changed. It is as simple as doing a string comparison between the previous response and the one you just received. If it did change, forward the new value to the target URL. If not, do nothing.

Store

For that matter, you need to store those responses. CaptainHook uses PostgreSQL because it’s what most developers are familiar with, but it could be any persistent storage.

You can store the raw body of the response in the DB directly. At the time of writing, I chose to compute the MD5 of the responses to store and compare shorter strings. After investigation, it seems like it doesn’t bring any advantage other than slightly reducing the storage in the DB. Computing the hash is also way more CPU intensive than comparing two longer strings.

Conclusion

That’s all you need to emulate webhooks on any API. It can be as simple as a simple block of code in your project handling just one endpoint, or a separate service like CaptainHook, working for more generic use cases. But keep in mind that this solution won’t replace real webhooks. It’s more like a workaround when none is available. It’s still doing a higher number of requests on the target API so use this approach carefully.

Happy Hooking!