A rather pointless performance test for Node

So today I performance a rather pointless performance test.

I was wondering what difference the version of Node makes to simple performance tests: I would expect that the newer the version, the faster (in requests per second) it would be.

Because it should be, right?

The Test

I copied-and-pasted a simple hello-world script...

const http = require('http');

const hostname = '127.0.0.1';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

and then installed Apache Bench, or ab to hammer it:
ab -c500 -n100000 http://localhost:3000/

But how to get the different versions? Node version management is a pain on Linux, it's a nightmare on Macs, and I've not tried it on Windows but I can't imagine that it's fun.

Here's the actual useful bit of this, an app called n. Just install it via sudo npm install -g n and you've suddenly got a very powerful tool for switching node versions.

So, if you want the LTS version?

 % sudo n lts

  installing : node-v12.19.0
       mkdir : /usr/local/n/versions/node/12.19.0
       fetch : https://nodejs.org/dist/v12.19.0/node-v12.19.0-linux-x64.tar.xz
   installed : v12.19.0 (with npm 6.14.8)

Easy.

The Results

Each test was ran three times, because #science, on the latest version of each major version back to Node 5. No changes where made in the code, or the way the test was run. This was all ran from my 8th gen i5 laptop:

version     req/s  1         2          3
14.13.1      9732.52     9866.98     9894.88
13.14.0     11065.33    12051.36    12247.49
12.19.0      9261.99     9926.34     9869.66
11.15.0     12510.51    13466.22    13783.50
10.22.1     11740.82    11925.83    11727.70
 9.11.2     12702.04    13667.15    13632.01
 8.17.0     10570.54    11579.70    11914.19
 7.10.1     11062.46    11622.44    11650.05
 6.17.1      9236.92     9293.16     9593.82
 5.12.0      8991.05     9790.51     9560.93

Interesting. From version 5,6, and 7 there's an obvious increase in performance from this simple metric. After that, into version 8 it starts to flatten out, then something odd happens for 12 and 14.

What does this test show us?

It shows us, in this very limited set of hardware, for a non-real-world code example, what the requests per second would be.

That's about it.

From this, we can't tell anything, really, about the major differences between the versions, or what that version would be like to run our code in production.

So this is a pointless set of numbers?

No. Well, yes, but the reason I did this was to try to disprove my bias to the notion that 'latest is greatest'. To really test if your code is going to be benefit from a bump in version numbers, you're going too have to understand a couple of things:

1. What your code does, and how the new version changes that
2. What the performance of your code looks like currently
3. What, other than performance, are the reasons not to upgrade.

Most of the time, if you're moving up between minor and bug versions, security is your biggest reason to upgrade. If there is a bug that particularly affects you then that's also another great reason to move up also.

So what now?

I'm currently looking at correctly measuring the performance of our production codebases with tools such as Gatling, and tracing the code to find other bottlenecks and memory issues. Once we understand what the current numbers are, we can then use those to compare and contrast before future upgrades.

We have a policy of sticking to the LTS (Long Term Support) version for production code. This comes after a range of versions of node was being used in different projects in our stack, causing all kinds of problems testing, deploying, developing etc. The tool n has helped with quickly changing versions, but if you have a single policy for a version, it makes things a bit simpler.

Cover image was Image by jacqueline macou from Pixabay