How to Connect to Private AWS Resources with SSH Tunnels and Bastion Hosts

The Problem with Publicly Accessible AWS Resources

When you first develop infrastructure for a new project, you naturally optimize for rapid development. You want to get something - anything - out the door, and you therefore want to be able to write code and debug issues quickly.

Because of that, it's awfully tempting to spin up servers and databases in public subnets so that you can readily connect to them for debugging sessions. It's nice to be able to ssh my-user@my-web-server to do some live code debugging, or psql -U my-user -h my-database-instance to assess the current state of your database.

Sure, AWS allows you to do that. You can expose SSH on your webservers and you can expose your AWS Relational Database Service (RDS) and Redis/ElastiCache servers to the internet for easy access. But should you? Exposing databases and servers publicly yields a pretty obvious security issue: your sensitive data is directly accessible to anyone on the internet.

You might argue "it's fine; we lock down our VPC security groups so that only people from our office IP can access our EC2, RDS, or ElastiCache instances. Other people on the internet are blocked because of their IP addresses". While that's true - locking down your resources this way is better than nothing - this strategy yields two pitfalls:

1. Allowing anyone from your office IP address to access your AWS resources is still a security issue. Should guests on your WiFi, or John the junior developer really have access to production databases? By locking down your resources by public IP, it's pretty hard to discriminate between DevOps users and the rest of your organization - everyone likely uses the same public IP address. Plus, as your organization grows, more attack vectors are created. A single compromised user on your network can wreak havoc on your production systems.
2. Your remote workers are going to struggle to hit resources. If your employees are working from home (which, in this COVID-riddled year, they very likely are), you will either need your employees to maintain a VPN connection to your office, or you will need to maintain a whitelist of all of your employees' home IP addresses. If your employees happen respond to an outage from an airport or coffee shop, they might spend minutes (hours?) manually monkeying with VPC security groups before they can even get access to address the outage.

This is a problem that I've had to solve everywhere that I've worked, so I imagine it's a problem others have had to face, too. Let's look at how to do it right, optimizing for both security and accessibility at the same time.

Enter AWS Systems Manager

Let's suppose you've followed security protocols, and have placed your various AWS resources (EC2, RDS, ElastiCache, etc.) into private subnets and removed public access to them. How can you securely grant access to members on your team (like your senior devs and DevOps) who need them? AWS provides Systems Manager to let you inspect and access your AWS resources - even those residing in private subnets. With Systems Manager, users exchange their AWS credentials for temporary shell access to EC2s. They can get that access both on the CLI, and through the AWS web console.

To enable access to your EC2s, simply install the AWS SSM agent onto the servers that you spin up. What's the deal with the extraneous "S" in "SSM", you might ask? It's historical - Systems Manager used to be called "Simple Systems Manager" and the extra "S" stuck - just go with it. Good news - if you're building your servers off of Amazon Linux, macOS, Ubuntu Server, or some Amazon-provided Windows Server images, the SSM agent is already installed for you - you just need to turn on a machine and grant it a specific role defined in AWS's docs.

Once you have an EC2 with the SSM agent installed, you can open a shell into your instance with the aws CLI tool:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-0bf6ff366c16d651f
sh-4.2$ whoami
ssm-user
sh-4.2$ cat /etc/system-release
Amazon Linux release 2 (Karoo)

When you have the SSM agent installed, you don't need to worry about juggling IP whitelists. Your systems management teams can access resources from anywhere (the office, at home, or even a coffee shop) using their AWS credentials. And, without AWS credentials John the junior dev has no access to production databases. Sorry, John!

What About Accessing RDS or ElastiCache?

With AWS Systems Manager you can remote into EC2s, but what about databases like PostgreSQL/RDS or Redis/ElastiCache? AWS doesn't allow you to directly SSH into the systems running RDS or ElastiCache. Instead, I suggest spinning up a minimal EC2 instance called a bastion in your VPC that you can remote into with Systems Manager. You can remote into the bastion, and once there you can access your databases.

Make sure that you assign your bastion a VPC security group, and create ingress rules so that your RDS and ElastiCache security groups allow access from your bastion security group. Once you have all that set up, you can access your databases from the command line on the bastion:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-048a3bf2269ad7caf
sh-4.2$ psql -h 10.0.2.88 -U testuser postgres
Password for user testuser:
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

postgres=> CREATE TABLE users (id INT, email VARCHAR(40));
CREATE TABLE

You might ask, "can't I just spin up a bastion in a public subnet, and SSH into it?" Yes and no. Yes, you can totally do that. No, it's not a good idea - that opens up the same security vulnerabilities that leaving your servers and databases in public subnets does.

Tunnels... Tunnels Everywhere

At this point, we have options for remoting in to EC2s directly through Systems Manager, and we can access our databases indirectly via a bastion EC2. The last thing I want to look at is leveraging SSH tunnels through the bastion to access database resources as though they're locally accessible. That way, you can connect your favorite database GUI, like pgAdmin to your RDS instances.

First, we need to get an SSH public key into our bastion. I'll assume you already have a key in $HOME/.ssh/id_rsa.pub. We'll use the aws CLI to temporarily load up that key for user ssm-user into our bastion:

$ aws ec2-instance-connect \
    send-ssh-public-key \
    --availability-zone us-east-1a \
    --instance-id i-0b6c737cc21dc01a9 \
    --instance-os-user ssm-user \
    --ssh-public-key file://$HOME/.ssh/id_rsa.pub
{
    "RequestId": "f8f3db2a-3107-4c0a-ae3d-94e2d7fff620",
    "Success": true
}

Next, I'm going to modify my laptop's $HOME/.ssh/config file so that servers starting with i-will proxy their connections through the aws ssm start-session command we used earlier. I'll add:

host i-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Now, I can simply ssh ssm-user@INSTANCE-ID to access my instance:

$ ssh ssm-user@i-0b6c737cc21dc01a9
Last login: Tue Dec 15 20:27:34 2020 from localhost

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ssm-user@ip-10-0-0-71 ~]$ ls
file.txt system.log
[ssm-user@ip-10-0-0-71 ~]$

Now that I have SSH access to my bastion (even though it's in a private network that's not accessible from the internet!) I can spin up SSH tunnels like I would with any other SSH connection. To create a tunnel to my RDS instance, for example, I can simply run:

$ ssh ssm-user@i-0b6c737cc21dc01a9 -NL 5000:10.0.2.88:5432

And then from a separate shell I can access my database "locally" on port 5000:

$ psql -h localhost -p 5000 -U testuser postgres
Password for user testuser:
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> \dt
         List of relations
 Schema | Name  | Type  |  Owner
--------+-------+-------+----------
 public | users | table | testuser
(1 row)

Cool, huh? Using tunnels through your bastion you can access any resources that reside in your private network without needing to expose them to hooligans on the internet... or John the junior dev.