Building a Simple Bot Protection With NGINX JavaScript Module (NJS) and TypeScript

Cover Photo by Phillip Glickman on Unsplash

I love Lua. I also love NGINX. The three of us get along just great. Like every relationship, we’ve had our highs and lows (yes, I’m looking at you Lua patterns), but overall life was perfect. Then, NGINX JavaScript Module (NJS for short) came along.

NGINX JavaScript module was first introduced in 2015 but recently received a big boost in functionality with the 0.5.x update. Since I'm a sucker for anything JS, I decided to test it out by building a simple (read naive and not production ready ) bot protection module 🤖.

Configuring NGINX

Before diving into bot fight, we have to set up NGINX to support the JavaScript module. The instructions below are for my setup (Ubuntu 20.4/Nginx 1.18), so YMMV, but the general idea should be the same for most setups.

1. Start by adding the NGINX PPA key by running:
   
   curl -s https://nginx.org/keys/nginx_signing.key | sudo apt-key add -

2. Setup the repository key by running:
   

sudo sh -c 'echo "deb http://nginx.org/packages/ubuntu/ focal nginx" >> /etc/apt/sources.list.d/nginx.list'

1. Update the repository list by runningsudo apt update.

2. Install NJS by running sudo apt install nginx-module-njs.

If all went well, at this point, you should get this lovely message on your terminal:

1. Enable NJS by adding the following to the top of your main nginx.conf file:

load_module modules/ngx_http_js_module.so;

1. Restart NGINX to load NJS into the running instance:

sudo nginx -s reload

Now your NGINX is ready for some JS love, so let’s move on and create our first line of defense — IP filtering!

Opening Act — Creating the Project

Our bot protection project is going to be written in TypeScript. For that, we need to create a project that will transpile TypeScript to ES5 JavaScript, which NJS can understand. As you may have guessed, NodeJS is a must here, so make sure you are all set up before continuing.

1. Create the new project folder and initialize it:

mkdir njs-bot-protection && cd njs-bot-protection
npm init -y

1. Install the required packages:

npm i -D @rollup/plugin-typescript @types/node njs-types rollup typescript

1. Add the build script to the package.json ’s scripts section:

{
    ...
    "scripts": {
        "build": "rollup -c"
    },
    ...
}

1. To compile the project, you’ll need to tell the TypeScript compiler how to do that with the tsconfig.json file. Create a new tsconfig.json file in the root of the project, and add the following content to it:

1. Lastly, let’s add the rollup config, which will wrap everything up and produce the endgame js file that NJS will read. Create a new rollup.config.js file in the root of the project, and add the following content to it:

And with that, our boilerplate is all loaded and ready to go. That means it’s time to kick some bots!

Round 1 — IP Filtering

Our first line of bot defense is IP blocking; we compare the IP of an incoming request with a list of known IPs with bad reputations, and if we find a match, we redirect the request to a “block” page.

We’ll begin with creating the JavaScript module:

1. In the project root folder, create a new folder called src, and then inside of it create a new bot.ts file.
2. Add the following code snippet to bot.ts :

💡 So what do we have here?

• Line 1 : Imports the built-in module for the file system (i.e., fs). This module deals with the file system, allowing us to read and write files, among other activities.
• Line 2 : Calls the loadFile function, passing it the name of the file we wish to load.
• Lines 4–12 : The implementation of loadFile. First, we initialize the data variable to an empty string array (line 5), then we try to read and parse a text file containing a list of bad IP addresses into the data object (line 7), and finally we return the data object (line 11).
• Lines 14–21 : The implementation of verifyIP — the heart of our module (for now). This is the function we will expose to NGINX to verify the IP. We first check if the array of bad reputation IPs contains the current request client IP (line 15). If yes, redirect the request to the block page and end processing (lines 16 and 17). If not , redirect internally to the pages location (line 20).
• Line 23 : Exports (read exposes) verifyIPexternally.

1. Build the module by running npm run build in your terminal. If all goes well, you should find the compiled bot.js file in the dist folder 🎉

With the file in hand, let’s configure NGINX to be able to use it:

1. In your NGINX folder ( /etc/nginx in my case) create a folder named njs and copy bot.js from the previous section inside it.
2. Create a new folder called njs under /var/lib , create a file called ips.txt inside it, and populate it with a list of bad reputation IPs (one IP per line). You can either add your own list of IPs or use something like https://github.com/stamparm/ipsum.
3. In your nginx.conf , under the http section, add the following:

js_path "/etc/nginx/njs/";
js_import bot.js;

💡 So what do we have here?

• js_path  — Sets the path for the NJS modules folder.
• js_import  — Imports a module from the NJS modules folder. If not specified, the imported module namespace will be determined by the file name (in our case, bot)

1. Under the server section (mine is on /etc/nginx/conf.d/default.conf ) modify the / location as follows:

location / {
    js_content bot.verifyIP;
}

By calling verifyIP using the js_content directive we set it as the content handler, which means verifyIP can control the content we send back to the caller (in our case, either show a block page or pass the request to the origin)

1. Still under the server section, add the block.html location and the pages named location:

location [@pages](http://twitter.com/pages) {
    root /usr/share/nginx/html;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}

location /block.html {
    root /usr/share/nginx/html;
}

(The namedpages location will be used by our NJS module to internally redirect the request if it shouldn't be blocked. You likely have your own logic for this redirection so change this to fit your needs)

1. At the bottom of the file, add the server block for port 8080:

server {
        listen 8080;
        location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
    }
}

1. Under the /usr/share/nginx/html folder, add the block.html file as follows:

And with that, our IP protection is ready! Add your own IP to the ips.txt file and restart NGINX (sudo nginx -s reload). Browse to your instance and you should be greeted with the following:

Round 2 — JavaScript Detection

Our second protection layer is JavaScript detection. We use this detection to determine if the visitor coming to our site is running JavaScript (which every normal browser should do) or not (a warning sign that this visitor might not be a legitimate user). We begin with injecting a JavaScript snippet to the pages that will bake a cookie on the root path:

1. Add the following code snippets to bot.ts :

💡 So what do we have here?

• Line 1 : Imports the built-in Crypto module. This module deals with cryptography, and we will soon use it for creating an HMAC.
• Lines 5–18 : The implementation of getCookiePayload. The function sets a date object to one hour ahead of the current time (lines 6–8), then uses the date object to HMAC (using the crypto module) the signature we passed to the function (the value object) with the date object (lines 10–14). Lastly, the function returns the cookie information in a string format (name, value, expiration, etc.). You may notice the cookie value contains not only the hashed signature but also the date object we used to HMAC the signature with. You’ll see why we do that soon.
• Lines 20–30 : The implementation of addSnippet. The function buffers the request data, and once it finishes (line 23) it:
• Creates a signature based on the client IP and the User-Agent header (line 24).
• Replaces the closing head tag with a script section that inserts a cookie (from the getCookiePayload function) on the browser side using JavaScript’s document.cookie property. (lines 25–28).
• Sends the modified response back to the client (line 29).

1. Export the new addSnippet function by updating the export statement at the bottom of the file:

export default { verifyIP, addSnippet };

1. Under the @pages location block, modify the / location as follows:

location [@pages](http://twitter.com/pages) {
    js_body_filter bot.addSnippet;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}

Unlike verifyIP, we don't want addSnippet to manage the content of the response, we want it to inject content (a script tag in our case) to whatever response comes back from the origin. This is where js_body_filter comes into play. Using the js_body_filter directive we tell NJS that the function we provide will modify the original response from the origin and return it once finished.

1. Restart NGINX and browse to a page on your instance. You should see our new script added just before the closing head tag:

If the client is running JavaScript, a new cookie called njs will be baked. Next, let’s create the validation for this cookie/lack of cookie:

1. Add the verifyCookie function (and its supportive functions/variables), to bot.ts :

💡 So what do we have here?

• Lines 5–11 : The implementation of the updateFile function, which uses the fs module to save an array of strings to a file.
• Lines 13–52 : The motherload implementation. When validating the njs cookie, we have a flow of verification and consequences we have to follow:

a. We begin with extracting the njs cookie from the request’s Cookie header (lines 14–20).

b. If we don’t have a cookie (or we do and it’s malformed), we compare the client IP against our list of client IPs that have reached us without a cookie. If we find a match from within the last hour, we fail the request (returning false, lines 26–27). If we don’t, we delete the IP (if it's on the list but past one hour) and pass the request (lines 29–34).

c. If we do have a cookie, we split it into a timestamp and a payload and use the timestamp to create our own HMAC hash based on the request’s User-Agent header and client IP. If our own HMAC matches the HMAC of the njs cookie, we pass the request. Otherwise, we fail it (lines 38–45).

d. If anything goes wrong during the validation, we fail open (meaning pass) the request (lines 48–51).

1. Add the new verify function, which calls the new verifyCookie function, and act according to its result:

🔥 At the point you might be thinking to yourself at this point that this verify function looks eerily similar to the verifyIP function from the earlier — you are absolutely right, and I will touch on that in a minute!

1. To test our new cookie validation functionality, open up your configuration file (mine is at /etc/nginx/conf.d/default.conf ) and change the js_content directive from verifyIP to verify:

location / {
    js_content bot.verify;
}

1. Restart NGINX and try to visit the site twice without the njs cookie — ✋ 🎤- you are blocked!

Final Round — Bringing It All Together

So now we have the cookie verification, but we took off our IP verification because we can only have one js_content directive, how do we go around fixing that?

You may remember that a few minutes ago we created the verify function (which eagle-eyed readers may have noticed is VERY similar to the verifyIP function we used before). If we update our verifyIP function so that it returns a boolean response as verification, and add that verification to verify, we get the best of both worlds with one big function that verifies requests for both IPs and cookies!

1. Refactor the verifyIP function as follows:

1. Update the verify function to call verifyIP as follows:

1. Update the export statement, as we no longer need to expose verifyIP:

export default { addSnippet, verify };

1. Restart NGINX and enjoy your home-made bot protection using NJS and TypeScript 🎉

🍾 The module source code is available on GitHub!