RetroFeed Work Log #2 - Basic Package Setup

We left off in Idea & Basic Setup with a project idea and a general tech plan. The current and next two articles will cover a bunch of foundation-level issues for any Internet application project. More precisely we’re going to:

• Setup a repository with a basic structure, establish various processes and link the some developer-oriented services. This will be covered in this article.
• Setup the application part of the project. We’ll deal with the web framework, setting up sessions, migrations, authentication, Docker, server-side rendering etc. There’s a lot of meaty stuff here, and even if the application itself won’t do that much at the end, a lot of the bricks will have been laid. This will be covered in the next article.
• Setup the infrastructure part of the project. We’ll deal with setting up GCP, constructing a “runtime environment” for deployment and building actual build pipeline to make it happen. This will be covered in the third and last article of this section.

At the end of the process we’ll basically have a very simple web application, with probably two screens - a “hello world” main page, and a “logged in” page, showing the name of the user. However I’m expecting all of the processes & foundational issues to be decided and solved. From how to run tests locally, to monitoring or rolling back a bad release. As mentioned in the previous article, expect modern tools, but conservative choices.

Since that’s out of the way, let’s start with the proper “part one”.

You can check out the code here. It is hosted on GitHub and it’s a single repository. It’s tagged as v0.0.2. I’ll keep referencing exact versions in the future as well, rather than just any commit.

Recapping the last post, the main architectural choices are:

The software stack is going to be Node with TypeScript for the backend, NestJs with Express as a web framework, Raynor with Raynor RPC for NestJs for RPC-ish type stuff. I’ll use Postgres as a datastore with Redis for caching. I’ll use Knex.js as a data access library. I’ll use React & it’s ecosystem for the frontend bits. The local environment will be run off Docker, with docker-compose. Deployment will be done on Google’s cloud. I’ll use the managed SQL and Redis offerings, and run Docker in production. But I won’t go the Kubernetes route, and simply run the services on a compute node with docker run. Infrastructure as code via terraform, of course. Other services I’ll be using are GitHub for code hosting, Travis.Ci for builds, NPM for package hosting, Codecov.io for coverage reports, Cypress.io for end to end testing, Auth0 for easy authentication, and GitHub’s API for repository stuff. The code’s going to be released under the MIT license and I’ll try to be as open as possible about the things I’m doing.

A bunch of these can already be seen in the repository at this stage. The file structure looks something like this:

.
├── .gitignore
├── .nvmrc
├── .travis.yml
├── LICENSE.txt
├── README.md
├── docker-compose.yml
├── package.json
├── scripts
│ ├── lint.sh
│ ├── send-coverage.sh
│ ├── serve-watch.sh
│ ├── serve.sh
│ ├── test-watch.sh
│ └── test.sh
├── src
│ └── index.ts
├── test
│ └── index-test.ts
├── tsconfig.json
├── tslint.json
├── tsnyc.json
└── yarn.lock

There’s not much to it in fact LOC wise. .gitignore, README.md and LICENSE.txtare pretty boilerplate. We use yarn as a package manager, tslint with it’s default and quite strict ruleset, istanbul/nyc for coverage reports etc.

I’ll highlight some of the more interesting files and choices, but feel free to poke around to see how things tick.

The package.json looks something like this:

{
    "name": "retrofeed",
    "version": "0.0.2",
    "description": "Manage updates for your project's dependencies",
    "author": "Horia Coman horia141@gmail.com",
    "scripts": {
        "serve": "./scripts/serve.sh",
        "serve:watch": "./scripts/serve-watch.sh",
        "test": "./scripts/test.sh",
        "test:watch": "./scripts/test-watch.sh",
        "lint": "./scripts/lint.sh"
    },
    "devDependencies": // Not shown
    "dependencies": // Not shown
}

The scripts section is interesting. It only references the ./scripts directory. This is where I hold all of the actions one can do as a developer. And ideally, everything you’d want to do - from running a test, to doing a deploy, to regenerating LetsEncrypt certificates will have a script here. The current structure is:

scripts/
├── lint.sh
├── send-coverage.sh
├── serve-watch.sh
├── serve.sh
├── test-watch.sh
└── test.sh

Notice that not all of them are mapped to the scripts key. So you can say yarn lint, but not yarn send-coverage. This is intentional. It highlights the ones you should be using locally. And while you could run the others, those are meant for the CI server or the production environment. Conversely however, there shouldn’t be anything you’d want to run which wouldn’t have a script.

For example, serve-watch.sh looks like (a bit after v0.0.2):

#!/bin/bash

set -e # Fail on first command

source scripts/setup-env.sh # Sets up environment variables

npx ts-node-dev --inspect=${INSPECT_PORT} -- src/index.ts # Does the action

Notice that npx is needed, cause the script is meant to also be run outside of yarn serve:watch.

The code itself is minimal. There’s just one src/index.ts file with a single function, and a corresponding test/index-test.ts file with tests for it. We use mocha, chai, sinon etc for testing, since these are quite standard in the JavaScript world. More code is going to come later.

A thing to notice is that there’s no compile.sh or build.sh script. TypeScript tooling is good enough that there’s a equivalents of node (ts-node) or node-dev (ts-node-dev) which allow skipping the classical compilation step entirely. So this gets rid of a lot of complexity and can’t really see a downside to it – perhaps performance in production. But I doubt it.

One interesting thing is the set of integrations we have. They’re in turn:

• Travis.CI as a build tool. Every commit in every branch results in a build. But only tagged commits with a version number get pushed to npm. Checkout the .travis.yml for the build configuration. It’s pretty standard - some setup, a run of yarn lint and yarn test, a test success action of sending coverage results to coveralls, and a tagged deploy action of pushing to NPM.
• NPM as a package repository. Packages are pushed automatically from Travis, on a green and tagged build.
• Coveralls.io for managing coverage reports. There’s a bit of magic involved in getting the right data with the setup. Every successful build send coverage data, and at some point tests are going to fail if coverage isn’t good enough.
• Cypress.io for end to end testing. This integration isn’t yet visible here, since there’s nothing to end to end test.
• Ngrok for making the current server instance available publicly. Again, this isn’t setup here yet, but it’s just a neat tool.

Anyway, that’s it for now. As I’ve said, not much happening, but a lot of preparation for it to happen. In the next parts we’re going to actually have something serving as a web application, and a full cloud deployment for it. See you soon!

Checkout all the articles in the series here.