The Road More Followed

If you've ever searched for a solution to the timeless, but never-quite-satisfactorily-answered problem of how to configure your Node backend with secrets and other values, you've doubtlessly seen the dotenv or config libraries. These libraries makes it dead easy to get up and running. Simply add a rule to .gitignore to keep your .env file out of source control, and pull values from it into your app code using environment variables. In this post I'm going to show you a Better™ way.

"Better" I hear you say! What presumption! "Better is completely subjective!" OK OK get back down off the high horse. Here's my working definition of Better.

Better === More Flexibility && More Predictability && More Access Safety

Now that we have that out of the way, let's get into it. I know your PM could pop by any moment. 🧐

Flexible Config

So what's the issue with using environment-based libraries to pass in API keys, tokens, or feature flags? Well, when your app is small, nothing! If you only change 1 or 2 values when you push to prod then you are likely going to be fine using environment-based configuration. However as your app scales and you add more features, services, and complexity, managing things this way will become problematic.

For instance, let's imagine your app uses some transactional mailing as part of it's functionality. When you are running locally you probably don't want to be sending off tons of mails to fake addresses (or even real ones), which might degrade your sender reputation or chew up API credits.

Since our app is small, let's just add a conditional around our API call to check for the environment and skip them locally.

if(process.env.NODE_ENV !== 'production'){
    console.log('[ MAILER ] Skipping mail in development', mailPayload)
} else {
    return MailerService.sendMail(mailPayload);
}

Cool! So now we won't send mails unless we're on prod. Easy as pie.

[meanwhile on slack]
Product Manager: Hey bud! Can we test the new email templates? Please send one of each mail to dev@genster.cz thanks!

Hmmm ok. So how can we solve this... We could set NODE_ENV to production, and trigger the mails, but that would also connect to the prod DB, and... oh, maybe that new pricing algo would get invoked as well since it uses a similar env flag... I guess I'll have to edit the app code to flip that logic temporarily, and hopefully remember to change it back again after!

Sound familiar? Don't lie.

When you hang lots of functionality off of the running app environment, you couple together many factors in ways that are not always easy to reason about.

A more flexible tack would be to create a feature flag for these types of functionalities.

First we'll add a new flag to our .env file

transactionalEmailsEnabled=false

Then we use this flag to control emailing rather than the running environment. By doing this we create a flexible configuration system that is much more scalable, and gives you granular control from outside of application code. Ideally all flags should be independent of all other flags so that none of them rely on the state of others to function. Some exceptions might be an on-off flag, and an API key for that feature. Use your brain to discover more exceptions :)

Sidenote: Devops people love this as they can test various feature permutations without having to dig into your Beautiful App Code, and without bugging you when your Beautiful App Code is not perfectly clear.

If we're using the popular dotenv lib then we can edit our .env file with these values. If we're using the config lib, we can add a local.json or local.yaml file to add some value overrides. Editing a few lines in these files to toggle behavior is a snap, but doing this a lot, or testing groups of things together becomes a bit hairier. I don't know about you, but my brain just won't remember which of 20 flags should be on and off for a specific test. At least not for very long. To make this process easier to manage, we'd need a way to have multiple versions of our config file and tell the app which to load.

A great way to do this is with command line flags. By default, dotenv will only load the one .env file. It does however have a way to point it to a different file.

(from the docs)

node your_app.js dotenv_config_path=/custom/path/to/.env

Alriiiight. Now we can have more than 1 .env file, and can load in which config we want! The downside here is that dotenv will only load 1 file. That means each variant you want has to have all the app values in it. It's all or nothing. When you add new ones, don't forget to add them to all files!

The config lib is better in this regard. It will always load default.json or default.yaml, and also load another file (either the matching environment file, or local.json) and basically do Object.assign(default, environment) giving you the ability to only have overrides in your secondary file. However config has a major downside. With this lib, you're basically screwed when you want to manually load a specific file. It only loads files based on the current NODE_ENV value, which is a real bummer.

Predictable Config

When you stop using process.env.NODE_ENV in your code, you gain much more of an understanding of what your app is doing, and what it will do when you deploy it. Instead of having 35 environment-based logic branches in your app, you only need look into your loaded config files to know what is and what isn't switched on.

No more surprises when your app does something weird on prod that you never saw it do in test, or staging.

No more having to maintain a convention of if(process.env.NODE_ENV === 'production'), or was it if(process.env.NODE_ENV !== 'production')? 🤔 Those are totally different things, and it will bite you!!

Safer Config

About a year ago I switched from using .env files to using the config library. The main reason was config's .get() and .has() methods.

The .get() method will try to load the value, and if the value is missing will throw an error and crash your app. Everyone hates app crashes, but everyone hates magical javascript runtime errors even more! If a required value is missing, the app should not start. Period.

The .has() method will check for the value but will return a boolean rather than throwing an error. This can be used to check for an API key, and if missing only log those API call payloads as well as add a log message that the service is disabled and why for debugging. As a rule I log out the status of all configurable services when the app starts.

The other advantage that config has over dotenv is the fact that values are encapsulated rather than stored in a global variable. "Global variables?! This is Node, not a browser!" Well, process.env is a global namespace just the same as window is in browser-land. Why do we get all mushy about let and so religious about using global variables only to use them at the very heart of our backend apps? Just like global variables, anything can change these values. Don't tell me you've never spent 40 minutes tracking down some magical bug which turned out to be the fact that you accidentally wrote if(checkDidPass = true)? Mmmm Hmmm. process.env values are no different.

By choosing a configuration library that uses getter methods rather than direct property access, you ensure values never change once your app is up and running.

Better Config

An ideal configuration library would allow the following functionalities.

1. Ability to load default values in any format (json, yaml, envfile, js exports)
2. Ability to load in an override file to change selected default values
3. Ability to manually select this override file from anywhere on disk (or maybe even remotely!)
4. Accessing nonexistent values should throw helpful errors
5. Config values should be impossible (or difficult) to change after initial load

Surprisingly enough, this ideal library does not exist! The functionality described here it actually pretty simple however. In fact after I overcame my shock at the lack of a good and simple configuration management library, I just wrote one myself. If there's interest, I can publish it on NPM (never done that before!).

Here's what it boils down to.

const fs = require('fs');
const path = require('path');
const yargs = require('yargs');
const yaml = require('js-yaml');
const _ = require('lodash');

// configDir is separate from configFile as we also load other files like certificates from the same location
let configDir = typeof yargs.argv['config-dir'] !== 'undefined' ? yargs.argv['config-dir'] : false;
// configFile should be located inside of configDir
let configFile = typeof yargs.argv['config-file'] !== 'undefined' ? yargs.argv['config-file'] : false;


/**
 * Reads cli arguments and loads in config files
 * 
 * @returns Configuration Object
 */
function createConfigurationMap() {
  let fullConfig = {};

  // always load these defaults from within the app
  let defaultConfig = yaml.safeLoad(fs.readFileSync(path.join(__dirname, '../config/default.yaml'), 'utf8'));
  _.merge(fullConfig, defaultConfig);

  if (configDir && configFile) {
    if (/^..\//.test(configDir)) configDir = path.join(__dirname, configDir);
    let overrideConfig = yaml.safeLoad(fs.readFileSync(path.join(configDir, configFile), 'utf8'));
    _.merge(fullConfig, overrideConfig);
  }

  return fullConfig;
}

/**
 * This class gets instantiated with a configuration object, 
 * and exposes the get() and has() methods.
 * 
 * It does not contain the value-reading code to make it easy to pass in mock values for testing
 *
 * @class CMP_Config
 */
class CMP_Config {
  constructor({ CMP_ConfigurationMap }) {
    this.configurationMap = CMP_ConfigurationMap;
  }

  has(prop) {
    let val = this._resolvePath(prop);
    return val !== undefined;
  }

  get(prop) {
    let val = this._resolvePath(prop);
    if (val === undefined) throw new TypeError(`Value for ${prop} is missing from config.`);
    return val;
  }

  loadCert(certName) {
    let certDir = configDir || path.join(__dirname, '../config');
    return fs.readFileSync(path.join(certDir, certName), 'utf8');
  }

  _resolvePath(path) {
    return path.split('.').reduce((o, p) => (o ? o[p] : undefined), this.configurationMap);
  }
}

module.exports = {
  CMP_Config,
  createConfigurationMap
};

This code is just what we use at Genster, and not really flexible enough to be an NPM module quite yet. In our case we have the file loading, and the actual class separated so as to make testing with mock values easy. You can instantiate the config class with any object, rather than having to load things from a file.

We use it as a module inside an Awilix DI container, but you could also use it like const config = CMP_Config(createConfigurationMap()). Just ensure that the module you have it in is a singleton and not reading in the config file dozens of times :D

To make this really easy to work with, we have our default.yaml file checked into git, containing dummy values for all but the most trivial services. Then we have a rule in .gitignore which allows you to have local copies of override files without getting them tracked by accident.

config/override-*

Additionally I've created a few different start commands in package.json to make working with these overrides really easy. This let's us run against a staging DB, or enable all third-party services. The override files just get shared directly with developers who need them via secure direct messaging.

{
  "scripts": {
    "devbe-staging-db": "nodemon app.js --config-dir=../config --config-file=staging-db.yaml",
    "devbe-services": "nodemon app.js --config-dir=../config --config-file=config-with-services.yaml"
  }
}

Hopefully this will help out some people suffering from similar pain that we had a few months back. There are a lot of posts about managing app configs floating around, but many of them have less-than-ideal solutions and none of them contain much by way of real-world use cases and complexities. In another post I'll cover how we manage getting config values into staging and production environments using Ansible.

Thanks for reading!