Essential Principles for Seamless Updates, Robust Execution, and Uninterrupted Performance

There are things that inevitably will happen to the application you are creating:

1. It will be updated (often)
2. Its data scheme will change
3. It will fail during execution or deployment
4. Any external resource it uses will become unavailable for some period of time (disc, network, underlying hardware, another service, or database)

It’s important to keep that in mind during the development.

There are a few principles I urge to follow in order to have applications that are easy to maintain, test, deploy, troubleshoot, and develop.

In general, I recommend following The twelve-factor app methodology. By following it, most of the principles described above will be satisfied automatically. I won't detail them all but here are a few I want to highlight:

Design for automation

Every application will be used in automation, so make sure it is designed for that as well. For example, keep configuration in the environment variables instead of the configuration file, explicitly define dependencies, do not hard-code addresses/passwords/ports, etc.

Keep the state out of the application

Also known as the “Process disposability principle”. Containerized applications must keep their states externalized or distributed and redundant. In addition, the application must be quick to start and to shut down, and even be ready for a sudden, complete hardware failure.

If you need to store data - store it in a separate service, do not use local storage for that matter. Local storage may be used for the generated data, like cache. It is always a good idea to use a clear interface to work with data.

Life-cycle conformance (Design for Failure)

The application should be able to detect and gracefully shutdown on SIGTERM (finish current request, close opened connections to the DB, revert transaction, refuse any incoming request, etc). The shutdown should be as quick as possible - Kubernetes will detach pod from endpoint preventing any new incoming requests and wait 30 seconds by default to pod gracefully terminate before forcibly kill it with SIGKILL.

Another aspect of life-cycle conformance is to be able to switch back to the previous version of the application if necessary (basically support future-compatibility for the one version at least). In practice do not migrate databases in the way that the previous version of the application can not use it anymore (the same with libraries, environment variables, API, etc). The simplest way is to use conversions, versification, and deprecation mechanisms available in your framework/language ecosystem.

Image immutability/Self-containment

Containers are meant to be the same across all environments and behave the same does not matter when they run. To keep it immutable, please:

• Do not install/update libraries on container start
• Do not change environment variables like PATH etc.
• Container configuration should be done on the build stage. Everything else should be done by the configuration.

Observability

The application should report its status and the process state. Meaning it has to have:

• Health check (Liveness probe) endpoint to determine that application started successfully. Typically something like /health with response 200 on success.
• Operation check (Readiness probe) endpoint to determine that application are ready to receive data. Typically something like /status or /read with response 200 on success.
• Metrics endpoint to publish metrics for monitoring. I recommend using Prometheus exposure formats. An endpoint could be exposed by a separate port or path.
• Logs that clearly report operations performed. Logs should be a first-class citizen in the code. Log statement should at least include time, log level, severity, and the meaningful logging message. Try to avoid multi-line messages. It’s a good idea to use JSON as a format for logs.

Store configuration in the environment

An application and its configuration should be completely independent. Further, storing configs constantly in code should be avoided entirely. The simplest way to do so is to build configuration from the environment variables. It's a good idea to have a separate internal interface for configuration in the application to avoid configuration cache and have an opportunity to have one configuration source even in case of changing storing configuration in the environment variables to the dynamic configuration backed by configuration system like Zookeeper. Additionally, pass all required parameters through configuration instead of application CLI parameters if it's not mandatory.

Scale-out via the process model

In order to add more capacity (start additional processes on additional machines), your application should expect to add more instances instead of more memory or CPU on the local machine. So it’s better to design an application to have multiple instances of it where every instance uses 1 CPU than develop an internal multi-processing system to utilize multiple CPUs on the machine by one instance.

Treat logs as event streams

Meaning you should stream logs to a chosen location — not simply dump them into a log file. I recommend using stdout by default.