So, you want to hear another story, eh? One filled with DevOps and DevOps-ey things? Ha! I've got a story for you. This is a tale of key vaults, automation, and getting things done.
The Vault
It starts with an Azure Key Vault. This is a native service that stores application secrets. Traditionally the application.config or web.config, for a service or website respectively, contains connection strings that your code requires in order to run. This is fine, however doing this is equivalent to writing your passwords on a post-it note stuck somewhere underneath your keyboard (I'm looking at you, Jerry!). If anyone gets to your desk they'll know where to look for it, and once they grab your connection strings it's game over.
To combat this, folks will often encrypt it. The drawback to this though is that the encryption key is often on the machine, which isn't much better. If the Nuget package or whatever mechanism you are using to decrypt it at runtime isn't kept updated, your application is likely to break because it can't read the config file, and it leaves you with what looks like Matrix code in your files (this has happened to me, and it's terrifying). The best place to go then is Azure Key Vaults (AKV) to store them in a specially encrypted external database which allows for not only credential-based authentication, but use of a unique-to-the-individual SSL certificate for authentication to avoid having to write your password somewhere and remember it.
Empowering Mercury
One of our development teams(Mercury) had recently switched to using AKV for something new that they were creating. They needed to add a lot of secrets to their new vaults and it's quite honestly very tedious. The portal generally allows you to upload 1 at a time, and it's a lot of clicking around. Since you're likely copy-pasting all the values in, it's easy to copy the wrong one in. So I gave them a workaround I had used on previous projects:
Since it's possible to use special characters in keys and secrets, and secrets can be very long, such as a JSON object, it's not a great idea to have them be paired in a 2D array, or to try and split a string based on a unique characters.
The general guidance that I give my coworkers is to make sure you verify the secrets beforehand for validity to ensure that every key has a secret, as secrets are not allowed to be blank (rule of Azure). If there was a mistake they could easily fix the value and re-run this, and if there was a scenario where a secret was given the wrong value, it would just be given a new one and only the new version would be active. They now had one of my favorite scripts, and things were fine for a time.
Bad Moon On the Rise
Happily armed with their new tool, the Devs got to work. Over the weeks that followed I received requests during the day to upload new secrets to production, as our Devs are not allowed direct change access to those resources. Once I even came in at night after hours to help with a deploy. It was about this time that all of us began to notice the issue: Me.
It wasn't that I was a problem, or intentionally holding them back, but rather I was inadvertently holding them back from deploying more often than not because I would usually be in a meeting or otherwise predisposed with something else. Sometimes we realized that the key vault values needed to be changed because they weren't what we were expecting a few hours later, and I'd need to step in. Every minute I wasn't available to them was a minute they had to wait to fix the issue.
Since this was a brand-new process, we didn't have a lot of existing guidelines in place because we didn't know how frequently Devs would ask us to make the updates.
The entire point of DevOps is to be empowering and hands-free, and this was turning into something akin to what large companies have with CAB meetings, and the like. None of us expected this simple process to turn out to be this complicated. It had become clear to everyone involved that we needed to implement a better workflow for this, pronto.
Enter the Octopus
The suggestion came up that we automate this to avoid direct DevOps intercession. Agreeing, I did what I should have done at the start - added the script from above as an executable step within Octopus Deploy to allow updating the AKV. It fit directly into their other deploy procedures, and now Dev is able to deploy key vault updates as they need without worrying about who on DevOps is there to assist. This ended a cycle of what was becoming a 3x-a-week-request-with-the-occasional-wait pain point and turned it into a no waiting, self serve request submission.
What are the morals of this story?
Automation does not solve all of your problems.
It sometimes is not enough to script out the task to make it less tedious. Just because we can does not mean that we should.
There are times when a process still needs to have automation driving it forward so that it doesn't hold up production, and there are times we are better off not automating things. This is a nuanced process that requires a formula with non-static variables to get close to right: (saved_cost_of_task - time_spent_troubleshooting) > cost_of_task. The amount of time saved by having the automated task combined with the time spent troubleshooting should be greater than the cost of not automating the task itself. If a task saves you a lot of time but you waste more time troubleshooting it than you do having it run, because it's a complicated process that requires human thought (and assuming you did this in an optimal manner) you're likely wasting more time than you'd be saving by automating it.
As an example, I could automate Team City and Octopus to deploy every single SQL script to run on all databases after it successfully goes to production. While this saves time, it wastes a lot of effort troubleshooting: There's a lot of permutations of scenarios I would have to capture with a deploy process for SQL where the order that I deploy one more more apps in a specific order. The amount of knowledge of the unique scenarios places an unfair burden on Dev, and becomes a detrimental necessity for DevOps to be available to assist. I could write an entirely separate post about this, but I think you get my point.
Be prepared to redefine the process.
We can't always foresee where or when we will need automation. We need to be willing to periodically take a step back and re-assess our processes and re-align our thinking so that we can make the right decisions.
As time progresses and you grow as a team, the willingness to change needs to exist. Before we started this journey down Key Vault Lane, updates to existing keyvaults were rare and infrequent. As we built more of the application, it became apparent that we had outgrown the process, and like an old pair of pants that no longer fits, we had to go get a new one that fits our needs and still make us look good.
Top comments (0)