Why syncing .env files doesn’t scale for secrets management

Learn why using a Universal Secrets Platform is the key to managing environment variables at scale and eliminates the need for syncing .env files.

The benefits of using environment variables to keep secrets out of source code are well established. But are .env files the best method for managing them?

Secrets management has evolved beyond the limited Key-Value storage that .env files provide. However, most developers are either unaware of .env file’s shortcomings or have simply become numb to the pain from the many years of usage and lack of innovation.

This post aims to highlight the risks of continuing to use .env files, why the major cloud vendors and hosting platforms offer a built-in secrets or environment variables store which can be used instead, and how secrets managers such as Doppler and HashiCorp Vault are providing the much-needed management and secrets automation layer on top of encrypted and access controlled secret storage.

A brief history of .env files

The usage of environment variables and .env files for application config and secrets largely began around 2013. It was a long overdue and important step towards more secure secrets management practices.

Libraries such as Python’s dotenv and Node’s dotenv made it easy for developers to use .env files and environment variables for application configuration, giving developers a simple path to removing secrets from their source code for good.

To think that .env file usage has remained practically unchanged for over eight years is quite remarkable. It should come as no surprise then, that it’s time to say goodbye to .env files in exchange for alternatives that better meet the needs of modern application development.

The problems with .env files

Using .env files allowed us to move secrets out of source code. Unfortunately, they introduced a new set of challenges:

• Scaling issues related to syncing .env file changes across environments and different cloud providers, increasing the risk of infrastructure misconfiguration and potential downtime.
  
  

• Easy for .env files to contain syntax errors, requiring additional tools such as dotenv-linter to be added to pro-commit hooks or GitHub checks.
  
  

• Sharing of unencrypted secrets in .env files over Slack when secrets change or new developers join a team risks breaking the principle of least privilege by exposing secrets to potentially unauthorized users.
  
  

• Inconsistent format of environment variables can cause issues, e.g. Docker and GitHub require unquoted values while other packages do not.
  
  

• Patchy and inconsistent support for multi-line secrets such as TLS certificates, SSH keys, JSON, and YAML.
  
  

• Secrets used in multiple applications must be duplicated in every .env file (instead of dynamic secrets referencing), making updating and rolling credentials tedious and repetitive.
  
  

• If persisted to disk in plain text, they may be readable by unauthorized users with access to the system and threat actors if file restrictive file permissions aren't used.
  
  

• Easy to accidentally expose to malicious bots if placed in the webroot of a web server or S3 buckets.
  
  

• Local development environments break whenever team members forget to share updates that need to be applied to their .env files, e.g. when a feature branch is merged that requires a new secret.

It's clear .env files have serious implications for application security Next, we’ll take a closer look at why the productivity impacts of using .env files may be worse than you think.

The hidden productivity costs from using .env files

Small repetitive problems, such as manually updating .env files across several servers as part of the deployment process, while perhaps initially frustrating and annoying, can easily become just an expected part of the application deployment lifecycle.

While some developers would argue that the papercuts associated with using .env files are minor, one thing we can all agree on is that interruptions can have serious productivity implications for writing code.

According to a recent study, the average lost time per serious interruption is 23 minutes:

"You need to get into the mindset for development and then slowly trace back to where you left off. This can easily take more than 30 minutes." - Gloria Mark, Professor of Informatics California, Irvine.

The cost of misconfiguration errors is not just the time spent fixing an .env file related issue. It's the impact of unexpected context switching and the challenge of getting back into a state of deep work, also known as “flow”.

Why developers have ignored traditional secrets managers

Traditional secrets managers such as Azure Key Vault or AWS Secrets Manager provide encrypted storage and fine-grained access controls, specially designed for storing secrets such as API keys, database credentials, SSH keys, and TLS certificates.

They are incredibly secure, robust, and enterprise-ready. But unfortunately, secret managers such as HashiCorp Vault are built for security teams, not developers.

As a result, they can be complex to implement correctly and often require secret-fetching implementation details to leak into application code—the exact opposite of the benefits afforded by using language-agnostic environment variables.

Even security-minded developers motivated to use a traditional secrets managers have typically given up for one primary reason: Using .env files was much easier.

Instead of environment variables, a vendor-specific SDK, platform integration, or custom application code for fetching secrets from a vendor’s API is often required.

For example, take this AWS Secrets Manager SDK for Node.js sample code for fetching secrets:

// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "<<{{MyRegionName}}>>",
    secretName = "<<{{MySecretName}}>>",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});

// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.

client.getSecretValue({SecretId: secretName}, function(err, data) {
    if (err) {
        if (err.code === 'DecryptionFailureException')
            // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InternalServiceErrorException')
            // An error occurred on the server side.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidParameterException')
            // You provided an invalid value for a parameter.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidRequestException')
            // You provided a parameter value that is not valid for the current state of the resource.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'ResourceNotFoundException')
            // We can't find the resource that you asked for.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
    }
    else {
        // Decrypts secret using the associated KMS CMK.
        // Depending on whether the secret is a string or binary, one of these fields will be populated.
        if ('SecretString' in data) {
            secret = data.SecretString;
        } else {
            let buff = new Buffer(data.SecretBinary, 'base64');
            decodedBinarySecret = buff.toString('ascii');
        }
    }

    // Your code goes here. 
});

It's this level of complexity compared with using environment variables that turns most developers off from the start.

As product teams are incentivized to ship software and new features as fast as possible, migrating to a traditional secrets manager usually only occurs due to regulatory requirements or security mandates.

But is it possible to still use environment variables for modern applications without .env files?

Modern platforms with native environment variable storage

Modern hosting platforms such as Netlify, Vercel, DigitalOcean, Cloudflare Workers, Fly.io, and Railway all come with secure environment variable storage built-in.

This not only shows how easy it is to migrate away from .env files but confirms that environment variables are still the best language and platform-agnostic method for injecting secrets into an application.

Do you need .env files for local development?

It may seem we’re still reliant on .env files for local development if hosting platforms only manage environment variables for applications running on their infrastructure. But this is beginning to change.

Every developer understands that inconsistencies between local and production environments are a recipe for unexpected issues. That’s why a USP provides first-class support for managing secrets in every environment. A trend modern hosting providers are also beginning to follow.

Vercel, for example, offers environment variable storage specifically for local development that is fetched and injected into the Node.js application via the Vercel CLI:

vercel dev

But what about developers using hosting providers without such functionality? This is where a USP such as Doppler fills in the gaps, eliminating the need for manually managed .env files.

Once developers have created a project and installed the Doppler CLI, secrets can be injected as environment variables into any application process:

doppler run -- npm run firebase-local-functions

Developer tooling is rapidly improving to provide a better integrated local development experience that will eliminate the differences between local and production environments and the need for manually managed .env files on developer machines.

A Universal approach to taming secret sprawl

Taming secret sprawl is a growing challenge for every development team and one that is only made worse as the number of .env files increase. We need an entirely new approach to secrets management that goes beyond incremental improvements—A Universal Secrets Platform.

Taking a “Universal” approach means being able to manage and sync secrets to every application on any platform by avoiding the problems associated with siloed secrets and antiquated solutions that don’t scale such as trying to sync dotenv files across platforms.

This can be achieved through a hub-and-spoke model where the USP acts as a single source of truth for secret storage and management with integrations automatically syncing secrets when they change to any external platform, including other secrets managers.

We hope our vision of a Universal Secrets Platform serves as inspiration for other secrets managers to create a more developer-friendly experience in order to make migrating away from .env files a more attractive option to developers.

Summary

We don’t need to sync .env files. We need the developer-specific workflows that a Universal Secrets Platform such as Doppler can provide.

The simplicity of .env files, while attractive at first, is also its greatest weakness. The demands of modern application development and the explosion of microservices across multiple clouds and platforms provide scalability challenges .env files simply can’t address.

The use of .env files was certainly an improvement over hard-coded secrets. But better options now exist for secrets management, and not only will your infrastructure be more secure without .env files, you’ll be more productive without them too.