Manage webhooks with MongoDB Functions and Triggers

In this post I explain how I built a webhook management system with MongoDB triggers and functions. This is part of mongodb atlas hackathon here in dev.to

Overview of My Submission

Webhooks are still a preferred choice of many organizations to communicate with 3rd party services. Since webhook calls are event-driven, if we use MongoDB Atlas as our database solution, we can use the features like triggers and functions to build a low code webhook management system. In this project, I showcase how to build such system with MongoDB Realm.

Submission Category:

Action Star

Link to Code

You can find the source code of the application at https://github.com/pubudusj/webhook-management-with-mongodb/

Setup instructions are available further in this post.

Additional Resources / Info

MongoDB features/tools used

1. MongoDB Atlas
2. Realm Triggers
3. Realm Functions
4. Realm External Dependancies
5. Realm CLI
6. Realm static hosting
7. MongoDB NodeJS SDK

Demo

You can see a fully functional demo at
https://webhookmanagement-gfbga.mongodbstitch.com/

Architecture

How it works

There are 3 parts to this implementation.

1. Webhook registration.
2. Webhook call for create candidate/user event.
3. View webhook call history.

Webhook Registration

1. There is an API exposed in the AWS API gateway which proxy to a Lambda function (RegisterWebhookFunction).
2. In this function, MongoDB NodeJS SDK is included, where it generate a webhook signing token and save the companyId, webhookUrl, event type as ‘candidate.created’ and the signing token to MongoDB collection - webhooks.
3. Generated webhook signing token will be returned from the API end point.
4. This token can be used to authenticate the incoming webhook.

Create user

1. There is an API exposed in the AWS API gateway which proxy to a Lambda function (CreateUserFunction).
2. Again, in this function, MongoDB NodeJS SDK is included, where it save user data including the companyId to a MongoDB collection - users.
3. In MongoDB Realm, I have created a application - WebhookManagement.
4. There, I have created a trigger (CandidateCreated), which will trigger when a new user is added to the users collection.
5. Also, there is a function created as CandidateCreated attached to this trigger, so this function will be executed upon the trigger.
6. Within this CandidateCreated function, there are few steps involved to call the webhook.
7. First it calls another function, GetWebhook to check if there are webhooks registered for this companyId and event type ‘candidate.created’.
8. If so, GetUserObject function will be called to fetch the respective user data from users collection.
9. With those information, there is another function - CreateWebhookPayload - called to create the payload for the webhook.
10. In this CreateWebhookPayload function, it generates a token to be added to the payload using signing token. To generate the token, I use SHA256 algorithm as: hmacSHA256(userId + currentTime, signingToken)
11. This token will be used to verify the authenticity of the request coming into the webhook. To generate this token, I use JS library 'crypto-js' which can be added easily into the realm app as an external dependency.
12. Once the payload is created, external url will be called including this payload using JS library 'axios'. Here also, I use external dependancy feature of realm app to load 'axios' library into the function.
13. Once the external api is called, the result will be saved into the webhook_calls collection using the function - CreateWebhookCall. Here, the url, payload, timestamp, status and response from the webhook call are saved into the webhook_calls collection.

View webhook call history

1. To fetch webhook call history, an API exposed in the AWS API gateway which proxy to a Lambda function (FetchWebhookHistoryFunction).
2. Given the companyId, using MongoDB NodeJS SDK, it will query the webhook_call collection to fetch the data.

How to Set up

1. First, create a new MongoDB Cluster with the name "Testing" at MongoDB Atlas.
2. Create a new user with username+password authentication at the Database Access section with permission read and write to any database.
3. In the Realm section, create a new application. Use name as "WebhookManagement" and choose the option: 'Use and existing MongoDB Atlas Data Source' and choose the cluster created in step 1 - Testing.
4. Under the 'Authentication' section, enable API keys and create a API key. Generated public key and private keys is required for realm cli authentication later on.
5. Install realm cli using npm install -g mongodb-realm-cli in your local machine.
6. Clone this github repository to your local machine.
7. This repo contains 3 main directories.
8. Goto the the realm directory, open the Makefile and set the values for parameters: REALM_APP_ID, APP_NAME, REALM_PUBLIC_KEY, REALM_PRIVATE_KEY.
9. Within the realm directory, run make realm_login, make realm_pull_app, make setup_functions, make setup_triggers, make deploy_all in the same order. This will create all the functions, triggers and dependancies in the Realm app.
10. backend directory contains a AWS SAM application to set up the required Lambda functions and API gateway apis which are consumed by the frontend app. Once the backend deployed, the API gateway url is required to set up frontend app. To deploy the backend, you need to have the connection string for the MongoDB database. You can get it from 'Connect' button on the dashboard of your DB cluster.
11. Instead of this AWS backend, you can use Realm endpoints as well. https://docs.mongodb.com/realm/endpoints/
12. frontend directory contains a VueJS app. Copy the .env.example file into .env and changed the VUE_APP_API_BASE_URL parameter with the one received in the backend deployment.
13. Then, use npm run serve for local testing or for production build, use npm run build --production.
14. Goto the hosting section of the Realm App and enable hosting. This will create a sample app with a url. ex: webhookmanagement-gfbga.mongodbstitch.com
15. Upload the frontend build files generated in step 12.
16. Confirm the changes and visit the url. You will see the frontend app.

Sample screenshot

Conclusion

Overall, Atlas and Realm provides great experience in terms of developer experience and performance. There are few thing I would like to see in future.

1. Retry feature: Currently, if Realm function fails, we cannot automatically retry.
2. Individual function calls in log: Currently if one function is being called within another function, it is not recorded in the logs. Only the calling function has a entry. Better to make all the function executions available in the logs.