How to write Scalable Nodejs Services [+Code Examples]

I recenlty had the opportunity to use OOP patterns on Nodejs environment, and I had a big time doing so.

Lets see what we’re gonna build this time:

• Build an API service that provides two resources: apps and developers , but more resources will be added in the future.
• Data is read from a NoSql database, but where to read the data may change in the future.
• When serving apps, we need to provide its associated developer on a special attribute “author_info” (every model may contain different rules which define how items are served).
• For now this service will be only responsible for reading data, but we should allow create and update operations on each model.
• We need to be able to change output format for particular models (we have partners that still work with xml format).

Model raw format as is stored on database is as follows:

developer

{
    "id": 23,
    "name": "AresGalaxy",
    "url": "https://aresgalaxy.io/"
},

app

{
    "id": 21824,
    "developer_id": 23,
    "title": "Ares",
    "version": "2.4.0",
    "url": "http://ares.en.softonic.com",
    "short_description": "Fast and unlimited P2P file sharing",
    "license": "Free (GPL)",
    "thumbnail": "https://screenshots.en.sftcdn.net/en/scrn/21000/21824/ares-14-100x100.png",
    "rating": 8,
    "total_downloads": "4741260",
    "compatible": [
      "Windows 2000",
      "Windows XP",
      "Windows Vista",
      "Windows 7",
      "Windows 8"
    ]
},

When fetching developers resource, it should remain as it is. But on fetching apps we need to merge developer model like this:

{
    "id": 21824,
    "developer_id": 23,
    "author_info": {
        "id": 23,
        "name": "AresGalaxy",
        "url": "https://aresgalaxy.io/"
    },
    "title": "Ares",
    "version": "2.4.0",
    "url": "http://ares.en.softonic.com",
    "short_description": "Fast and unlimited P2P file sharing",
    "license": "Free (GPL)",
    "thumbnail": "https://screenshots.en.sftcdn.net/en/scrn/21000/21824/ares-14-100x100.png",
    "rating": 8,
    "total_downloads": "4741260",
    "compatible": [
      "Windows 2000",
      "Windows XP",
      "Windows Vista",
      "Windows 7",
      "Windows 8"
    ]
},

So here are my thoughts on this:

We need to declare resources in a very straightforward manner, but it seems like every resource may be different, both in format and output.

So we need to extract “common” parts from Resource concept and build different and independent implementations for each Model.

What’s a Model ? On REST paradigm we usually call Resource to some domain item that is represented through an URL (api.io/rest/employee), we can easily interact with it using HTTP verbs and providing several parameters.

When writing maintainable APIs we need to differenciate from code which describes rules for every resource and code which defines how HTTP connections are fulfilled.

So I end up by creating two basic entities which are models and resources.

• Resources are classes which carry out the HTTP communication, for now we only have a single class because both apps and developers doesn’t contain differences at this layer.
• Models are classes which describe how operations, like reading data from database, joining data, formatting output, etc. is done for a particular domain entity, like developer and app, which are different and should be independent.

So we have two models classes developer and app and a single resource class. But, on runtime we’ve two resource instances, each of those has its model instance which is in charge of the specific domain rules.

So this is the start script:

const { setConfig } = require("ritley");
setConfig(require("./ritley.conf"));
const BasicResource = require("./resources/basic-resource");
[
  require("./models/app"),
  require("./models/developer"),
].forEach(Model => new BasicResource(new Model));

We’re using ritley. A lightweight package I made a month ago for fast backend development and specifically REST services.

So in the previous code we only require our ritley configuration which basically setups rest path, static assets folder (if needed) and the port to listen.

Then we just loop over models, and create a resource instance to be tied with its model and we’re up and ready.

Lets take a look over folder structure:

.
├── adapters
│ ├── low.conf.js
│ ├── low.js
│ └── low-provider.js
├── low.database.json
├── models
│ ├── app.js
│ ├── common.js
│ └── developer.js
├── package.json
├── README.md
├── resources
│ └── basic-resource.js
├── ritley.conf.js
├── run.js
├── test
│ └── developers.test.js
4 directories, 13 files

We’ve created models/common.js abstract class to be a starting point for further models:

const { inject, createClass } = require("kaop")
const LowProvider = require("../adapters/low-provider");
module.exports = CommonModel = createClass({
  adapter: null,
  constructor: [inject.args(LowProvider), function(_db) {
    this.adapter = _db;
  }],
  read() {
    return new Promise(resolve => resolve("read not implemented"));
  },
  find() {
    return new Promise(resolve => resolve("find not implemented"));
  },
  toString(obj) {
    return JSON.stringify(obj);
  }
});

You may noticed that I’m not using harmony ES classes. Thats because we need something like decorators and we don’t want to use any code transformer for now. Instead we’re using kaop to easily allow reflection techniques such as Dependency Injection.

So basically previous code declares an abstract model that will contain a lowdb instance adapter to access database. If we change our database service we only have to care about importing another provider.

Code below represents models/developer.js:

const { extend } = require("kaop");
const CommonModel = require("./common");
module.exports = DeveloperModel = extend(CommonModel, {
  path: "developer",
  read() {
    return new Promise(resolve =>
      resolve(this.adapter.getCollection("developers")));
  }
});

This only differs from common model on read method implementation, so we just replace it with a new one.

Note that our DeveloperModel contains path property which will be used by basic resource to listen several paths. Here is how:

const { extend, override } = require("kaop");
module.exports = BasicResource = extend(AbstractResource, {
  constructor: [override.implement, function(parent, _model) {
    parent(_model.path);
    this.model = _model;
  }],
  get(request, response) {
    let prom = null;
    if(request.query.id) {
      prom = this.model.find(request.query);
    } else {
      prom = this.model.read();
    }
    prom.then(result =>
      this.writeResponse(response, this.model.toString(result)));
  },
  writeResponse(response, body) {
    body && response.write(body);
    response.statusCode = 200;
    response.end();
  }
});

BasicResource extends from AbstractResource overriding its constructor for providing the path as you can see on highlighted line, which will be invoked for each instance. As we saw on the start script, models are passed down to resources to properly build our HTTP listeners. BasicResource’s get method will intercept all HTTP GET requests pointing to each path. One instance which was configured with developer model will effectively listen only on <host>/rest/developer path and so forth.

So, a client requesting <host>/rest/developer will be answered by BasicResource instance which was created with DeveloperModel instance.

For instance if we want to allow POST or PUT requests we need to write down a post method on BasicResource, ritley allow us to simply write methods named as HTTP verbs, so any requests that matches will be handled. If we need to allow POST only on several paths we may need to extend BasicResource into AdvancedResource or something which allows more HTTP verbs. This is best practices to properly separate concerns.

And perhaps models need to be grouped by what kind of resource they need to be mounted on.

For example:

const { setConfig } = require("ritley");
setConfig(require("./ritley.conf"));
const BasicResource = require("./resources/basic-resource");
const AdvancedResource = require("./resources/advanced-resource");
[
  require("./models/app"),
  require("./models/developer"),
].forEach(Model => new BasicResource(new Model));
[
  require("./models/company")
].forEach(Model => new AdvancedResource(new Model));

Now lets take a look over initial requirements to see if this is a good approach (question —  answer ):

• Build an API service that provides two models: apps and developers , but more resources will be added in the future. —  adding new models is pretty easy, we only have to create a new one by extending from common one, implement needed methods and declare on start script
• Data is read from a NoSql database, but where to read the data may change in the future. —  code responsible for accessing data services is on adapter folder, we’re using an awesome resource such as lowdb . We have 3 different files: low.conf.js which contains database path, low.js which wraps lowdb methods into domain related actions for models to consume and low-provider.js which declares a singleton dependency for injecting into models so we can rapidly switch over different database services :)
• When serving apps, we need to provide its associated developer on a special attribute “author_info” (every model may contain different rules which define how items are served). —  every model has its own methods for retrieving data down to resources so we can build de data as we fit. For this particular case, we created a method on the db adapter because nested models will be a pretty common case here adapters/low.js:

getMappedCollection(uid, joinuid, joinkey, newkey) {
  const joincollection = this.instance.get(joinuid);
  return this.instance
    .get(uid)
    .map(app => this.mergePredicate(
      app,
      joincollection.find({ "id": app[joinkey]}),
      newkey)
    )
    .value();
},
mergePredicate(app, subject, newkey) {
  return { ...app, { [newkey]: ...subject } };
},

and then, since app model is the only one who provides nested items we make use of it models/app.js:

read() {
  return new Promise(resolve =>
    resolve(this.adapter.getMappedCollection(
      "apps", "developers", "developer_id", "author_info")
    ));
},

• For now this service will be only responsible for reading data, but we should allow create and update operations on each model. —  already solved but gonna show an example:

const { extend, override } = require("kaop");
const BasicResource = require("./basic-resource");
// we only need to implement a new method since this class inherits
// from BasicResource
module.exports = AdvancedResource = extend(BasicResource, {
  post(request, response) {
    // create entry logic
  }
});

• We need to be able to change output format for particular models (we have partners that still work with xml format). —  if a particular model needs to output differently from others then we need to override toString() method from models/common.js . Say that DeveloperModel needs to output on XML format because some of our partners is still working with 2008 SQL Server so far..

const { extend } = require("kaop");
const CommonModel = require("./common");
const xmlconverter = require("awesomexmlparser");
module.exports = DeveloperModel = extend(CommonModel, {
  path: "developer",
  read() {
    return new Promise(resolve =>
      resolve(this.adapter.getCollection("developers")));
  },
  toString(obj) {
    return xmlconverter.stringify(obj);
  }
});

That’s all for today! Thank you so much :)

You can check the code here https://github.com/k1r0s/micro-ritley-lowdb-example