Three months ago we stopped using ENV files as the default export option in the Doppler CLI. This change led to a number of benefits including supporting multi-line variables and a deterministic schema. Before going too deep on the technical choices we made, let's first go over what ENV files are and how they're used.
What are ENV files?
ENV files are plain text files that store variables and secrets that you would not want hardcoded in your codebase. These variables could be a port number or a database url, and may change depending on where your code is deployed. For example, when developing locally you may use port 3000, but when deployed to Heroku your application will need to use the port it's dynamically assigned. An example ENV file when developing locally could look something like this with the schema of KEY=VALUE:
PORT="3000"
DATABASE_URL="psql://postgres@localhost/db_name"
AWS_KEY="rfiunb34fu93n49iufgn3o2o0ini2ef"
BUGSNAG_API_KEY="33dasdk34bsf23f35871as0fa27"
INTERCOM_KEY="ndeiovnkofrnij30490398u39RIBF39IOENIOWF3ENWKEWENVIO"
MAILGUN_KEY="fdjnbvjoenvlmswsdnjoelfv"
NEW_RELIC_LICENSE_KEY="b583d4eab955872122843a067faca9db5d4202af"
NEW_RELIC_LOG="stdout"
SALESFORCE_TOKEN="feojkvndfjkovnskfn3eni32one"
STRIPE_KEY="sk_test_ejidsfnvoiuebnfvoi3enjkdNCJSB"
Having the file is not enough though, you would also need a tool like foreman to parse the file and inject those variables into the environment.
Benefits
So what are some of the benefits of using an ENV file? Well these files live on your local machine which means you do not need a network connection to fetch your secrets. The schema is also quite simple so it's easy go into a file and add a new variable. Lastly, everyone knows this format so there is a ton of support by the open source community for parsers and managers.
Downsides
From our time working with ENV files and adding support for various use cases, we have found that there isn't a standardized schema all libraries use. For example, take a look at the sample ENV file below:
PORT = 3030
Notice that this ENV file that contains a spaces between the KEY and the VALUE. If we were to use bash to inject the variable into the environment with the source command we would get an error.
source spaces.env
# >> bash: PORT: command not found
Now if we use another tool like foreman we would see it parse without an error. This is because each library is deciding the schema of a ENV file instead of strictly following an open standard. These inconsistencies causes other problems to arise as well, such as parsing multi-line secrets. In this example a variable uses encoded newlines through \n:
CERT="-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\N-----END RSA PRIVATE KEY-----\n"
This newline is treated differently depending on which tool you use. Using the bash source command the \n in the string would not be converted to newline characters. On the other hand, using Python's most popular ENV library dotenv will convert the \n to newlines automatically. Now let's look at the inverse:
CERT="-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr
9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm
4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO
2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE
CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
OOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh
2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K
2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
ZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F
6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa
NxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
-----END RSA PRIVATE KEY-----"
In this example we have the same cert but with newline characters. Surprisingly the bash source command respects the newline character but the Node dotenv library does not. More interestingly is how the Node library breaks. It parses the value as "-----BEGIN RSA PRIVATE KEY----- and disregards all the other lines. I also find it funny that because it is a multi-line variable the quote detection algorithm broke, which can be seen by the first character being a quotation mark. If the quote detection algorithm was working correctly, you would see the value be stripped of it's quotation marks at the beginning and end of the string.
Alternatives
After realizing ENV files are problematic, we started looking at alternatives. We wanted something that has a universally accepted schema with no room for interpretation and a large community for support. The two data formats we focused on were YAML and JSON.
Let's start off with YAML. One of the primary advantages of YAML is that it is incredibly easy to read and write. It uses indentation and nesting as a way to designate structure. Let's look at a sample YAML file:
PORT: 3000
DATABASE_URL: "psql://postgres@localhost/db_name"
AWS_KEY: "rfiunb34fu93n49iufgn3o2o0ini2ef"
CERT: |
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr
9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm
4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO
2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE
CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
OOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh
2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K
2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
ZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F
6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa
NxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
-----END RSA PRIVATE KEY-----
At first glance the syntax looks very similar to the ENV format but when we look closer we see subtle differences. The YAML syntax uses colons instead of equal signs and has native support for multi-line strings. The one downside when using multi-line secrets is that indentation really matters. The fabled debates of how many spaces equals a tab come into play. With developers each having their own style, it can make YAML files prone to parsing errors when sharing.
JSON on the other hand has a wildly different syntax then YAML. Wikipedia has an accurate description of the language:
JavaScript Object Notation is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value).
Let's take a look at the same config of variables in JSON format:
{
"PORT": "3000",
"DATABASE_URL": "psql://postgres@localhost/db_name",
"AWS_KEY": "rfiunb34fu93n49iufgn3o2o0ini2ef",
"CERT": "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\n-----END RSA PRIVATE KEY-----\n"
}
One of the main beauties of JSON is that it is strictly enforced and there is only one way of accomplishing each task. For example, when we look at the variable PORT, we can see the value is wrapped in quotes to state it is a string. Unlike YAML, which will guess if the line should be cast to a string or number, JSON only has one way of notating strings and numbers. One other stark difference between YAML and JSON is how they handle multi-line variables. In JSON we can see it uses the encoded newline characters \n which we think is a safer bet than trusting humans with indentation.
Choosing JSON
We ended up going with JSON because it has a far stricter schema and has strong native support in most languages. After making the switch, we saw our customers' issues with parsing downloaded config files flat line. Since the Doppler CLI creates a fallback of your secrets by default when running your application, we decided to go one step further by enabling encryption by default.
We strongly believe that you are always going to be worse off having secrets on disk, but if you are going to, it is imperative that they are encrypted.
Doppler Ad
Tired of managing a fleet of secrets by hand using copy/paste? Want an end-to-end managed secrets manager that vaults all your secrets in one place, has built-in versioning and access control? Try out Doppler. It works great in local development (say goodbye to ENV files) and in production, plus it effortlessly scales with you as your team and products grow. Take a look at our quick install guide to see if it is a fit for your team.
Top comments (0)