EC2 Metadata Server and IAM Role Credentials

An IAM user is the starting point of the AWS account creation process. The root user is the first entity used to access an account. Users can be given a username and password which is a very familiar process for anyone who has accessed most login based websites. That said, the password is there until it's changed making it less ideal from a security standpoint for an EC2 instance accessing services. To avoid this an EC2 instance can have an instance profile which allows for a more secure access method.

Instance Profile

Simply put an IAM instance profile is a mapping of an EC2 instance to an IAM role. The role in question must authorize the EC2 service to access it or no access will be granted at all. However, the EC2 instance still needs some kind of authentication mechanism to validate that it should be accessing the services. What exactly is going on behind the scenes?

Metadata Server

It turns out that EC2 instances have access to something called a metadata server. It's known as IMDS (Instance Meta Data Service). The service itself is exposed to EC2 instances via the IP address 169.254.169.254 and runs as a simple HTTP service. There are two versions of the metadata service: IDMSv1 and IDMSv2. IMDSv1 was the original version and required no sort of authentication to retrieve data. IMDSv2 on the other hand uses a token system to authenticate against it. AWS recommends using IMDSv2 as it contains additional security features. Here is an example between a v1 and a v2 call:

IMDSv1

$ curl http://169.254.169.254/latest/meta-data/

IMDSv2

$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
$ curl http://169.254.169.254/latest/meta-data/profile -H "X-aws-ec2-metadata-token: $TOKEN"

As shown in the example v1 is a simple curl call while v2 reaches out to retrieve a time limited session token to make authenticated calls. The AWS SDK also comes with a handy command line tool called ec2-metadata which supports basic metadata information. Here is an example call that shows all the information it can get about an instance:

$ ec2-metadata --all
ami-id: ami-01107263728f3bef4
ami-launch-index: 0
ami-manifest-path: (unknown)
ancestor-ami-ids: not available
block-device-mapping: 
         ami: /dev/xvda
         root: /dev/xvda
instance-id: [redacted]
instance-type: t2.micro
local-hostname: ip-10-0-20-169.us-east-2.compute.internal
local-ipv4: 10.0.20.169
kernel-id: not available
placement: us-east-2b
partition: aws
product-codes: not available
public-hostname: [redacted]
public-ipv4: [redacted]
public-keys: 
not available
ramdisk-id: not available
reservation-id: r-09e1686225d2dc834
security-groups: launch-wizard-2
user-data: #!/bin/bash

echo "Hello World"
tags: 
not available

An interesting note is that the metadata server also allows for getting userdata which can then be used by any process to run the scripts. Major distributions will generally have cloud-init available to handle the pulling and execution of userdata.

Retrieving Credentials

It turns out that the metadata server is what provides the temporary credentials to the EC2 instance. It's not shown in the above attributes from ec2-metadata so it will have to be retrieve the curl way. Getting credentials requires the role name. This can be entered manually or simply retrieved from the metadata itself:

$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2023-05-25T23:23:41Z",
  "InstanceProfileArn" : "arn:aws:iam::[redacted]:instance-profile/TestRole",
  "InstanceProfileId" : "AIPARWUWYZG7GS35NWOIJ"
}

Then with the instance profile role name "TestRole" the credentials can be retrieved:

$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/TestRole
{
  "Code" : "Success",
  "LastUpdated" : "2023-05-25T23:23:40Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "[redacted]",
  "SecretAccessKey" : "[redacted]",
  "Token" : "[redacted]",
  "Expiration" : "2023-05-26T05:58:41Z"
}

The presence of "Token" shows that it's temporary credentials from STS (Security Token Service) letting authentication work without needing to store them on the EC2 instance.

SDK Support

So why AWS services allowed by the role accessible without having to worry about all this? That's because it's all handled behind the scenes by the various AWS SDKs available. The AWS CLI in particular runs off the python boto bindings. You can see here where it's fetching the credentials from the metadata server:

class InstanceMetadataFetcher(IMDSFetcher):
    _URL_PATH = 'latest/meta-data/iam/security-credentials/'
    _REQUIRED_CREDENTIAL_FIELDS = [
        'AccessKeyId',
        'SecretAccessKey',
        'Token',
        'Expiration',
    ]

    def retrieve_iam_role_credentials(self):
        try:
            token = self._fetch_metadata_token()
            role_name = self._get_iam_role(token)
            credentials = self._get_credentials(role_name, token)
            if self._contains_all_credential_fields(credentials):
                credentials = {
                    'role_name': role_name,
                    'access_key': credentials['AccessKeyId'],
                    'secret_key': credentials['SecretAccessKey'],
                    'token': credentials['Token'],
                    'expiry_time': credentials['Expiration'],
                }
                self._evaluate_expiration(credentials)
                return credentials

Source

This is then part of a chain of command credentials resolution:

    env_provider = EnvProvider()
    container_provider = ContainerProvider()
    instance_metadata_provider = InstanceMetadataProvider(
        iam_role_fetcher=InstanceMetadataFetcher(
            timeout=metadata_timeout,
            num_attempts=num_attempts,
            user_agent=session.user_agent(),
            config=imds_config,
        )
    )

Source

While environment variables are the first source, it's best practice to be using the temporary credentials provided by IAM roles and the metadata server. Also important to note that as the code suggests containers work slightly different in getting credentials. While the concept of getting them is the same the endpoints are different:

ENV_VAR = 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'
ENV_VAR_FULL = 'AWS_CONTAINER_CREDENTIALS_FULL_URI'
ENV_VAR_AUTH_TOKEN = 'AWS_CONTAINER_AUTHORIZATION_TOKEN'

Source

Conclusion

While most developers won't need to worry about this due to SDKs handling it automatically, it's still interesting to see how it works under the hood (and might help troubleshoot niche cases). It's considered best practice to use the temporary credentials this is all backing along with a principle of least privilege role attached to it. The metadata server may also provide useful information on non-credentials related metadata without having to rely on an AWS API call.