Keep up with the times: forget SSH, welcome AWS Session Manager

Although I’m a firm believer in Immutable Infrastructure, sometimes it’s just handy to quickly log into an instance to try something out or to do some troubleshooting. In that scenario, the easiest way to get remote shell access on Linux is through SSH (while its Window’s counterpart relies on RDP for that purpose). Technically this means that you need to allow inbound traffic on port 22 (for SSH) or 3389 (for RDP) on your remote host (if you use the standard ports).

Considerations

The problem I have with opening inbound ports on security groups is that this always involves risks. So whenever I can avoid opening an inbound port, I’ll prefer that solution. This article is about picking the best way for secure shell access to an AWS EC2 instance by offering a solution to enable Secure Access without opening any extra ports.

What are the options?

Before diving deeper into the topic let's go over the available options for Secure Shell access on AWS first. In this case, I see three options:

1. Good old vanilla SSH
2. EC2 Instance Connect
3. SSM Session Manager

In this regard: my main quest is to find the most secure option. At the same time, I would also like to find out each option's advantages and disadvantages. Something I really try to avoid is unpleasant surprises in the end. So let’s jump into the details of every option.

Option 1: Good old vanilla SSH or RDP

I won’t detail this option because I assume most of you are already familiar with it. The main pitfall to avoid is an insecure setup that allows root login, text password, etc. Even if correctly set up, I would consider Fail2Ban or sshguard as a minimum extra requirement. It’s the best-known option but also the one which is most under attack. Whenever possible I try to avoid exposing an SSH service to the world. Even correctly set up today, somebody could easily overwrite your hardened settings tomorrow. 😟

Option 2: Amazon EC2 Instance Connect

I consider EC2 Instance Connect as the new kid on the block. When compared to vanilla SSH, Instance Connect adds some nice security enhancements (Centralized access control, Short-lived keys, Auditability and Ubiquitous access). On top of these advantages, EC2 Connect is simple to use and offers an ‘out of the box’ solution that does not require extra configuration.

EC2 Instance Connect, however, does not answer some of my main security concerns: both a public IP and an open port to the world are still required. This also means it’s not an option for instances running in a private subnet. Furthermore, it offers limited OS support. Currently, only Amazon Linux 2 or Ubuntu (16.04 or later) are supported, and Windows support is completely left out.

Option 3: AWS Systems Manager Session Manager

The last option to scrutinize is AWS SSM Session Manager. It’s another AWS service to obtain secure shell access. The main benefits of Session Manager are:

• No open inbound ports and no need to manage bastion hosts or SSH keys, no public IP
• Instances in private subnets can be connected to, even without a NAT gateway present. VPC endpoints to Systems Manager are all that is required
• Centralized access control to instances using IAM policies
• Cross-platform support for both Windows and Linux
• One-click access to instances from the console and CLI
• Supports port forwarding
• Logging and auditing capabilities are provided through integration CloudWatch, CloudTrail, and S3

Furthermore, it supports both terminal access through the Amazon web console (SSM and EC2), AWS CLI, and vanilla SSH, although with the last option, you might lose some benefits (more on that later).

So we picked AWS Session Manager

Given all the above, we decided to go for AWS Session Manager as our preferred solution for secure shell access mainly because it’s the most secure pick out of the three, and it’s simple to set up.

Going a bit more in-depth

Installation

• Install SSM agent: given that Amazon Linux 2 is our preferred OS, it was easy to enable AWS Session Manager in our environment. That’s because SSM Agent is preinstalled, by default on Amazon Linux and a bunch of other AMI’s.
• The next step is to tweak your EC2 instance profile to allow instance access to Session Manager.
• You’re all set: meaning that you can create a secure shell connection from the CLI, the Web console, or even set up an SSH session tunnelled through Session Manager.

Logging, auditing, and encryption

When it comes to logging and auditing, it’s where AWS Session Manager really shines. In contrast to plain old SSH access where key pairs are often shared (although that shouldn’t be the case), a session initiated with AWS Session Manager is bound to a single IAM user. This way, every session is easy to trace back to a specific person. Also, CloudTrail keeps track of all API Calls made by Session Manager.

On top of that, AWS Session Manager offers the ability to store all session data (literally every single manipulation executed in the terminal, including its output) both on S3 and/or CloudWatch. Furthermore, both for S3 and CloudWatch, there’s the option to enable encryption, which I would recommend as a best practice. For my setup, I have picked the default SSE-S3 (AES-256) encryption for my bucket; CloudWatch logs are encrypted with KMS and a Customer Master Key (CMK).

For completeness, I include an encrypted S3 bucket example written in CloudFormation:

EncryptedBucket:
  Type: AWS::S3::Bucket
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    BucketName: !Sub security-logs-${AWS::AccountId}-${AWS::Region}-cfn
    PublicAccessBlockConfiguration:
      BlockPublicAcls: true
      BlockPublicPolicy: true
      IgnorePublicAcls: true
      RestrictPublicBuckets: true
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: AES256
    LifecycleConfiguration:
      Rules:
        - Id: clean-up
          Status: Enabled
          ExpirationInDays: 180

The last thing you want to do is monitor your SSM agent log files file, as explained here.

Can I finally log in, please?!

One of Session Manager's key advantages is its ability to provide SSH in different ways, each with its own advantages and disadvantages.

SSH using the AWS CLI

Secure Shell CLI access using AWS Session Manager is my personal favourite. It is achieved by using the following command:

> aws ssm start-session --target instance-id

Note: If you want to use the AWS CLI to start and end sessions that connect you to your managed instances, you must first install the Session Manager plugin on your local machine. The plugin can be installed on supported versions of Microsoft Windows, macOS, Linux, and Ubuntu Server.

SSH using the Web Console

There’s also the option to start a terminal session form within the Systems Manager console or the EC2 Console for less familiar people with the command line. It’s easy peasy, all you have to do is click a button, and a terminal will open within your browser.

What about secure copy?

One of the most common things done while using secure shell access is a secure copy (scp). This functionality, however, doesn’t come out-of-the-box when using aws ssm start-session. But don’t panic; to tackle this you simply can use S3 as a ‘proxy’. Since you seem to like using a terminal, you can use aws s3 cp to help you out. Just use an S3 bucket as a proxy to temporarily store your files (don’t forget to change your instance profile accordingly). The drawback is that this solution is a bit cumbersome, but it has the huge advantage that you don’t lose any of Session Manager's functionality.

Don’t open Pandora’s box

To wrap up, I need to mention that there’s also another possibility for copying over files. You can also enable SSH tunnelled connections through Session Manager. For me, that option feels a bit like Pandora’s box because logging and auditing are not available for Session Manager sessions that connect through port forwarding or SSH. This is because SSH encrypts all session data, and Session Manager only serves as a tunnel for SSH connections.

As mentioned before, logging and auditing are some of the major added values of using Sessions Manager; throwing it overboard doesn’t feel right. One could argue only to use it when scp is needed and to avoid it otherwise. History tells me that people just don’t work that way. I know when this option is available, people will just start using it to tunnel SSH sessions as well (and that includes myself 😄)

That is the main reason why I strongly advise to disable SSH connections through Session Manager. To achieve this, you need to add the following policy to a user or role.

{
 "Version": "2012-10-17",
 "Statement": [
    {     
     "Sid": "VisualEditor1",
     "Effect": "Deny",
     "Action": "ssm:StartSession",
     "Resource": "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
    }
]}

A little bonus

Another option is to tunnel good old SSH through Session Manager. Yes, that’s pretty nifty. It’s pretty easy too. To enable this, add the following to your .ssh/config.

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Port forwarding

You can also use Session Manager port forwarding to access your privately running RDS instance from your local machine.

Conclusion

Whenever you need secure shell access to an instance, I would recommend using AWS Session Manager. I would also advise you to disable SSH connections through Session Manager in favour of using an S3 bucket as a proxy to transfer files from and to your remote host.

Enjoy and until next time!