Deploying Concourse with concourse-up

There are a lot of Continuous integration tools out there, and they all have minor pros and cons but are generally the same. You push some code, that code triggers a build, and that build could trigger others. Sometimes these are called pipelines, but the only actual implementation of a pipeline build system I have seen is Concourse. There will be future posts on what a pipeline is and why it's such a powerful concept not that's not this post.

For this post, you should be up and running with Concourse in AWS with SSL and have your first pipeline in about 45 minutes, and most of that time will be waiting for AWS.

With this guide, you will

• Prepare AWS
• Provision valid SSL Certificates for free
• Deploy Concourse to AWS using concourse-up

You will need:

• Docker
• An internet connection
• An AWS Account with some credit
• A Domain with access to the DNS

Prepare AWS

You don't want to give concourse-up too many permissions so let's create some restricted keys and use those for our deployment.

First login to AWS and head to the IAM section.

If you're not familiar with AWS' IAM rules I recommend reading up on them. There are ton's of guides out there that can help.

I like to use Groups to manage Permissions so first create a group by going to the Group's section and then Add New Group. After naming your group, you will be prompted to attach security policies.

Add the following:

• AmazonRDSFullAccess
• AmazonEC2FullAccess
• IAMFullAccess
• AmazonS3FullAccess
• AmazonVPCFullAccess
• IAMUserSSHKeys
• AmazonRoute53FullAccess

You should see your new group with 0 users in the group list.

Now we need to add a user. In the Users section select Add User, name the user, and check the box Programmatic access. On the next section add the User to the group you created above. You don't need any tags so finish building the user.

This next screen is essential When you see Success click Download CSV and put it in a secure place.

Now you're ready to get your SSL Certs ready

Provision valid SSL Certificates

Concourse uses SSL for all of it's service to service communication. You also want to make sure that any content you serve is also SSL so let's do that next.

Certbot has a ton of options and can be really easy depending on your registrar and where you manage your DNS.
The most basic way is to use Certbot's manual image to generate your certificates.

sudo docker run -it --rm --name certbot \
    -v "$HOME/letsencrypt:/etc/letsencrypt" \
    -v "$HOME/letsencrypt:/var/lib/letsencrypt" \
    certbot/certbot certonly \
    --manual

Follow the prompts. You will be required to verify ownership of your domain. If you specify an exact domain like ci.focusedlabs.io you will need to serve a key at a specific domain. It will look something like this.

Create a file containing just this data:

6z1rbMF-R6XdSo-s8HrTMKEeLGIvszkIA1IJPqzr9iU.FUHXZSFi6r53NiLV9xzaHFovW4oXb9fjR0j9KNF36tw

And make it available on your web server at this URL:

http://ci.focusedlabs.io/.well-known/acme-challenge/6z1rbMF-R6XdSo-s8HrTMKEeLGIvszkIA1IJPqzr9iU

If you use a wildcard *.focusedlabs.io you will be required to add a TXT record to your DNS

Please deploy a DNS TXT record under the name
_acme-challenge.focusedlabs.io with the following value:

6gcFZfRzUVujnYkhMEF8po1i55e4NscuSFnUcnYR4CQ

Before continuing, verify the record is deployed.

After completing the verification, you will have the certificates in ~/letsencrypt/live

Deploy Concourse to AWS using concourse-up

Now we have certificates and AWS all ready to go so let's deploy Concourse.

Download the latest release of concourse-up from github https://github.com/EngineerBetter/concourse-up/releases and put it in your path.

Now in one simple command our concourse deploy will begin

concourse-up \
    deploy \
    --domain ci.focusedlabs.io \
    --tls-cert "$(cat $HOME/letsencrypt/live/focusedlabs.io/fullchain.pem)" \
    --tls-key "$(cat $HOME/letsencrypt/live/focusedlabs.io/privkey.pem)" \
    --region "us-east-1" \
    focusedci

Now watch the magic happen!

Next Steps

From here you can deploy a pipeline and start to experiment with all the community resources. Or you could set up a pipeline that will automatically refresh the certs with certbot.

Have fun and happy deploying!