Create Nginx extensions in JavaScript

If you need to adapt content, authorize, filter, change the behavior of Nginx, so njs can be the solution. Nginx proposes a JavaScript backend to manipulate requests and responses and even streams. This is njs and it is very powerful.

Nginx is a very good HTTP server and reverse proxy. It is simple to configure, easy to start, and it "does the job". We often use it as a proxy to backends. But, sometimes, you can feel some limitations. For example if you need to authorize users to several backends with a specific identity provider with a weird API. Or, sometimes, this is the backend which is problematic.

For example, I serve a Rancher backend, and Rancher provides its kubeconfig YAML file (dynamically generated for each user) with a certificate inside that I need to remove.

In short, I need to "filter" the content, because I cannot change the behavior of the backend.

And this is exactly when njs can be used!

What I could do is simply to detect that the user claims the KubeConfig file, remove the certificate entry in the YAML, and serve the response file. This is one of the vast variety of manipulation that you can do with njs.

What it njs?

njs is a JavaScript engine integrated to Nginx. njs provides a different approach that starts a JS VM on each needed process. It actually can use common JavaScript modules (like fs for example), and it's ES6 compliant. That means that you will code the scripts without the need of changing your habits.

What you can do

There are plenty of things that you can do with njs:

• Make authorization (with or without backend)
• Manipulate the output content, headers, status...
• Interact with streams
• Use cryptography
• And so on...

You must keep in mind that njs is not intend to create an application. Nginx is not an application server, it's a web server and a reverse proxy. So, it will not substitute a "real" web framework. But, it will help to fix some things that are hard to do.

Read this before!

Go to the documentation page here to check the global variables which are already defined by Nginx/njs, you'll see that there are many methods to trace, crypt, decode, or manipulate the requests.

Do not spend too much of time to read the page, but just take a look to be aware of the possibilities.

It's not activated by default

njs is not activated by default. It's a module that you need to load at startup.

The legacy method to activate it is to add load_module modules/ngx_http_js_module.so; on top of nginx.conf file.

For docker, you can change the "command" to force the module to be loaded:

docker run --rm -it nginx:alpine \
nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"

Then in your http services, you can now load a script and call functions for different situation.

Prepare the tests

To follow this article, and be able to test, we will start a Nginx service and a "Ghost" blog engine.

This will help to startup:

mkdir -p ~/Projects/nginx-tests
cd ~/Projects/nginx-tests
mkdir -p nginx/conf.d
touch nginx/conf.d/default.conf

cat << EOF | tee docker-compose.yaml
version: "3"
services:
  # our nginx reverse proxy
  # with njs activated
  http:
    image: nginx:alpine
    command: nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:z
    ports:
      - 80:80

  # a backend
  ghost:
    image: ghost
    depends_on:
      - http
    environment:
      url: http://example.localhost
EOF

Debug

Sometimes, your script fails to return somthing, you've got an error 500 and nothing is clearly displayd in the nginx logs.

What you can do is to add this in the "location":

error_log /var/log/nginx/error.log debug;

For docker/podman users, use:

# see the logs in real time
docker-compose logs -f http

And you can use in your script :

r.log("something here")
r.log(njs.dump(variable))

When you finally found the problems, you can then remove the "debug" suffix of the "error_log" directive.

There are many ways, many options, many situations

Nginx manages 2 types of configuration: http and stream. By chance, njs can be used for both.

In this article, we will only speak about the "http" mode.

When we work with "http" mode, we can tell to Nginx to use JavaScript to manipulate:

• The content (that means that we will generate the entire request before the call)
• The headers only (after request)
• The body only (after request)
• Set variables
• ...

The pipeline is not hard: import a javascript file, then use a Nginx directive to use a function for a situation. Nothing more.

First example, create a content

A first, let's create a file named example.js. To make it easy to test, put this in nginx/conf.d/example.js – in production environment, it's better to change the location of your scripts.

OK, so, this is the content:

function information(r) {
    r.return(200, JSON.stringify({
        "hello": "world",
    }));
}

// make the function(s) available
export default {information}

The r variable (argument of the function) is a "request" object provided by Nginx. There are many methods and properties that we can use, here we only use r.return().

It's now time to make the JavaScript function to be called as the content maker. In nginx/conf.d/default.conf, append this:

js_import /etc/nginx/conf.d/example.js;

server {
    listen 80;
    server_name example.localhost;
    location / {
        js_content example.information;
    }
}

Yes, that's all. We import the script, and we use the function as js_content. That means that Nginx will release the request to the script, and the script can yield the content.

Start (or restart) the container and hit the "example.localhost" domain:

$ docker-compose up -d
$ curl example.localhost
{"hello": "world"}

That's the first example. Here, we only generate a JSON output.

We can now do some nice things, like replacing the content.

Replacing the content

njs proposes several fetch APIs to get, sub-request or internally redirect the request.

In this example, we will replace the "/about/" page content by inserting a message inside.

The following configuration is not perfect (actually we could match location /about and call our JavaScript, but I want to show you the internal redirection) — but it will show you some cool stuffs that njs allows.

Change the nginx/conf.d/default.conf file with this:

js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # match everything
    location / {
        js_content example.replaceAboutPage;
    }

    # call the "ghost blog" backend
    # note the "arobase" that creates a named location
    location @ghost_backend {
        # important !
        subrequest_output_buffer_size 16k;

        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}

In this file, you have to pay attention on this:

We force "gunzip" to uncompress proxied responses. This is important if the backend forces gzip responses, and you want to make replacements.
We also make the "subrequest output buffer" size to 16k, you can set it to 128k if needed (for large CSS files if you subrequest them for example).

This is important because we will make subrequests later, and the buffer will be filled too fast

Then, in the example.js file, add the replaceAboutPage function and export it :

function information(r) {
  r.return(
    200,
    JSON.stringify({
      hello: "world",
    })
  );
}

async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.headersOut["content-type"] = "text/html";
    r.return(200, "Changed");
  } else {
    r.internalRedirect("@ghost_backend");
  }
}

export default { information, replaceAboutPage };

Take a minute to read the replaceAboutPage function. In this example, we only:

• return a page with "Changed" inside if the URI is "/about"
• either we use @ghost_backend location to proxy the blog

Restart the nginx container:

docker-compose restart http

And visit http://example.locahost. Then go to the "/about" page using the "About" link on top.

Nice, so now, we can do a better replacement.

We will need to "subrequest" the page. But a subrequest needs a "real URI path". So, let's add a location in default.conf first.

# a reversed uri.
# We remove the prefix and use the @ghost_backend
location /__reversed {
    internal;
    rewrite ^/__reversed/(.*)$ /$1 break;
    try_files $uri @ghost_backend;
}

Then, let's go to example.js and replace the replaceAboutPage function to this one:

//...
async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.subrequest(`/__reversed${r.uri}`) // call the reversed url
      .then((res) => {
        // copy the response headersOut
        Object.keys(res.headersOut).forEach((key) => {
          r.headersOut[key] = res.headersOut[key];
        });
        // replace the end of "header" tag to append a title
        const responseBuffer = res.responseBuffer
          .toString()
          .replace("</header>", "</header><h1>Reversed</h1>");
        r.return(res.status, responseBuffer);
      })
      .catch((err) => {
        r.log(err);
        r.return(500, "Internal Server Error");
      });
  } else {
    // in any other case
    r.internalRedirect("@ghost_backend");
  }
}
//...

One more time, restart http container:

docker-compose restart http

Then visit http://example.locahost/about/ – you should see:

Second method, use js_body_filter

A probably better solution is to use js_body_filter instead of js_content. This leaves Nginx makes the proxy pass, then we can manipulate the body.

So, let's change the default.conf file to this:

js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # call the "ghost blog" backend
    location / {
        js_body_filter example.replaceAboutPage;
        subrequest_output_buffer_size 16k;
        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header Accept-Encoding "";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}

Very important, here we force the Accept-Encoding to be empty, because Ghost will return a gzipped content that is impossible (at this time) to decompress from javascript.

Then change the JavaScript replaceAboutPage function to this:

async function replaceAboutPage(r, data, flags) {
  if (r.uri == "/about/") {
    r.sendBuffer(data.toString().replace("</header>", "</header>Reversed"), flags);
  } else {
    r.sendBuffer(data, flags);
  }
}

• r is the request
• data is the data to send, here it's the content taken from the Ghost backend
• flags is an object with "last" flags set to true or false

The function needs to use sendBuffer() to send the data to the client.

Of course there are many others things to do, like changing the "Content-Length" header, but this works.

It's very efficient and that's a good method to make some replacement, content checks or fixes to a response without the need to make a subrequest.

Make a fetch to outside

njs provides a global ngx variable. This object proposes the fetch API which is more or less the same as you can find in modern browsers.

Let's add a method to call the Dev.to API.

Please, do not abuse the API, it's a simple example, and you are pleased to not overload the servers

The following function will get the list of my articles.

async function getMetal3D(r) {
  ngx
    .fetch("https://dev.to/api/articles?username=metal3d", {
      headers: {
        "User-Agent": "Nginx",
      },
    })
    .then((res) => res.json())
    .then((response) => {
      const titles = response.map((article) => article.title);
      r.headersOut["Content-Type"] = "application/json";
      r.return(200, JSON.stringify(titles));
    })
    .catch((error) => {
      r.log(error);
    });
}

// don't forget to export functions
export default { information, replaceAboutPage, getMetal3D };

OK, now you think that you'll only need to add a js_content call inside the default.conf file. But... there will be some problems.

• ngx object need a "resolver" to know how to resolve the domain name
• also, you need to define where it can find certificate authorities

But, that's not so hard to do:

location /metal3d {
    resolver 9.9.9.9;
    js_fetch_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
    js_content example.getMetal3D;
}

So, one more time, restart the http container and call the /metal3d URI:

docker-compose restart http

curl http://example.localhost/metal3d

# response:

[

    "Python, the usefulness of \"dataclass\"",

    "Fixing a Promise return object to be typed in Typescript",

    "Flask / Quart - manage module loading and splitting",

    "Change local container http(s) reverse proxy to Pathwae"

]

Conclusion

Nginx proposes a very useful JavaScript extension system. It's easy to realize the large possibilities that can be made with it.

It's possible to create a custom cache system, to manipulate headers, reading token, validate authorizations, change the content, create a proxy on API, and many others things.

We only checked how to manipulate content, but it's also possible to manipulate "streams" with events.

Go to the documentation pages:

• NJS doc: https://nginx.org/en/docs/njs/
• JS object that you can use: https://nginx.org/en/docs/njs/reference.html
• Module directive:
  • http https://nginx.org/en/docs/http/ngx_http_js_module.html
  • stream https://nginx.org/en/docs/stream/ngx_stream_js_module.html
• Examples on GitHub: https://github.com/nginx/njs-examples/