This article is obsolete. Please follow this link to setup Vault on GKE
Introduction
Vault by Hashicorp is a highly scalable secrets store. Hashicorp the company is behind some amazing projects like Vagrant and Terraform among others. But as awesome as their products are, their documentation sometimes leaves much to be desired.
I was recently looking to get a Vault cluster up and running for a project I'm working on. This is the first time I had to set it up myself and realized that there are a whole bunch of gotchas that the documentation doesn't go into.
After hours of scouring the forums, github issues and such, I've finally managed to set it up. And I'd like to document the process. Let's jump in.
Goals
We are going to deploy a Vault cluster with Consul providing the High Availability. For this tutorial, I'll show how to deploy a private cluster behind a public load-balancer. Should be easy enough to switch to a fully private or fully public clusters from there.
My cloud provider of choice today is GCP (Do leave a comment if you'd like to see AWS or Azure specific versions and I'll see what I can do).
Pre-requisites
Step 1 - Terraform Modules
Head over to this github repo - https://github.com/hashicorp/terraform-google-vault and clone it to your local machine.
The good people at Hashicorp and Gruntwork have done most of the work for us. And the documentation is fairly extensive, but can still be overwhelming for someone new.
In the repo, we have a bunch of modules and a bunch of examples. The ones that are of interest to us are modules/private-tls-cert, examples/vault-consul-image and examples/vault-cluster-private-with-public-lb.
Step 2 - The Image
The TLS Certs
GCP does not support custom public images. So, we have to build our own. Let's head over to modules/private-tls-cert. Follow the instructions in the quick start section of the README.
Here are a few points to remember when filling out the variables.tf file.
Adding a default in HCL is by adding a default = "some value" line inside the variable block.
Th 'ca_public_key', 'public_key' & 'private_key' file paths can be any location on your system. But it's easier to put them right where we're going to use them. So, something like "../../examples/vault-consul-image/tls/{key}.pem" is a good way to go, where {key} refers to the three keys mntioned above. You can also put in any directory and manually copy the keys over.
The 'owner' variable is asking for the user who will own the keys on your local machine. The currently logged in user is a good default.
The way to fill list(string) variables is pretty much like a list of strings - ["some string", "some other string"].
The Packer Build
Once the tls certs have been gnerated, head over to examples/vault-consul-image. Ensure your TLS certs you generated are in the examples/vault-consul-image/tls folder. If you generated them at a different path, copy them over now.
Open the vault-consul.json file and set the following variables
- project_id - ID of your GCP Project
- zone - The GCP Zone whre you want to provision the vault cluster.
- ca_public_key_path, tls_public_key_path & tls_private_key_path - Paths to the TLS certs we generated in the previous section.
In the builders section of this file, you'll find that there are two images being built. One with Ubuntu 16.04 and the other with 18.04. If you're just going to use one of them, feel free to remove the other section.
Now, run packer build vault-consul.json.
When the build finishes, it will output the ID of the new Google Image. We'll need this for the next section.
Step 3 - Spinning up the cluster
Navigate to examples/vault-cluster-private-with-public-lb/ and open variables.tf & fill in all the variables.
Enter the id of the image Packer built for us in the previous step for both vault-source-image and consul-source-image.
Run terraform init
We need to make a slight change to one of the scripts. Stay with me. After running terraform init, a .modules folder will be created. Head over to examples/vault-cluster-private-with-public-lb/.terraform/modules/consul_cluster/modules/consul-cluster.
Find the line where it says instance_tmplate = .... and make the following edit
- Head back to examples/vault-cluster-private-with-public-lb/ and run terraform plan followed by terraform apply.
It'll take a while but once done, you have a brand new Vault Cluster provisioned and ready.
Step - Initializing, Unsealing and Using Vault
Here's fairly detailed documentation on how to init and unseal the vault cluster - https://github.com/hashicorp/terraform-google-vault/tree/master/modules/vault-cluster#how-do-you-use-the-vault-cluster
Follow along. Note that the init command needs to be run on only one of the nodes. The unseal command needs to be run with 3 different unseal keys on each of the nodes.
Setup an A record on a domain you own pointing to the load balancer (Terraform creates a load balancer with the 3 vault nodes added to it. They will fail health checks until you init and unseal them).
Now you can use the Vault UI at the domain name you've configured on port 8200 -> https://vault.example.com:8200/.
You will see that your browser thinks this is an unsecured connction. To overcome this, find the TLS certificate we created in step 2 and add the ca_public_key to the list of trusted CAs on your browser.
You can also access vault from the command line on your local machine. Just run export VAULT_ADDR="https://vault.example.com:8200 where vault.example.com is your vault domain/subdomain.
You'll get a similar "x509: certificate signed by unknown authority" error. This is fixed by adding the certificate to Apple Keychain. Just open Keychain Access and drag the certificates in.
Here's some info on how to do the same on Ubuntu - https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate
With that, you're all set.
P.S. I'm super new to writing technical tutorials. I'd love any feedback. Too verbose? Something unclear? Just let me know in the comments and I'll try to get them fixed up as soon as I can.
Top comments (0)