Caching SSM Parameter Store values in Lambda

Intro

This article is inspired by a blog post written by Yan Cui where he shows how we can cache AWS SSM Parameter Store values.

This approach is recommended so that we can re-use the same values during multiple invocations of our Lambda function and save costs of API calls to SSM Parameter Store.

The following is an example using AWS CDK where we define an SSM Parameter and use the aws-sdk to fetch the parameter and cache it for a given amount of time which we can configure.

Here's the repository for those who want to dive right in :)

Let's start by defining the parameter that we are going to fetch in our Lambda:

// lib/lambda-ssm-stack.ts
import * as ssm from '@aws-cdk/aws-ssm'

new ssm.StringParameter(this, 'dev-key1', {
  parameterName: '/dev/key1',
  stringValue: 'value1',
})

This is a snippet that creates a StringParameter i.e. an un-encrypted value to store with the parameter path /dev/key1.

In a project, it's recommended to store keys in a path based format like ${projectName}/${environment}/${keyName} so that we can easily fetch all the values under the same path. For e.g. myProject/dev/DB_HOST and myProject/prod/DB_HOST for different environments.

Next, let's define our Lambda function:

// lib/lambda-ssm-stack.ts
import { join } from 'path'
import * as ln from '@aws-cdk/aws-lambda-nodejs'

const lambdaDir = join(__dirname, '..', 'lambda-fns')

const handler = new ln.NodejsFunction(this, 'fetchParams', {
  runtime: lambda.Runtime.NODEJS_12_X,
  memorySize: 512,
  handler: 'handler',
  entry: join(lambdaDir, 'src', 'index.ts'),
  depsLockFilePath: join(lambdaDir, 'yarn.lock'),
  nodeModules: ['ms'],
  sourceMap: true,
})

In the above snippet, we have created a Lambda function using the Node.js 12 runtime allocating it a memory of 512 MB with some specific values. Let's have a look at these.

• The entry option accepts a file in which we will define our code to fetch the above defined parameter. In this case, it's an index.ts file in a directory named lambda-fns which we will look at further.

• The handler option is the name of the function which is expored in our index.ts file and that will be used by Lambda.

• The depsLockFilePath, nodeModules, and sourceMap options are for bundling our Node.js function before deploying. This is done using the aws-lambda-nodejs package that transpiles our TypeScript code to JavaScript with node_modules so that it can run it on Lambda.

We're done creating the function. Let's look at the final snippet of code required to deploy our stack:

// lib/lambda-ssm-stack.ts
handler.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['ssm:GetParametersByPath'],
    resources: [`arn:aws:ssm:${this.region}:*:parameter/dev*`],
  })
)

This final snippet allows the Lambda function access to fetch the created parameter from the Parameter Store, and not just any parameter but the one starting with /dev. This makes sure that we do not fetch any unwanted/unneeded secret values especially for other projects.

Now let's look at our Lambda function in the lambda-fns folder:

// lambda-fns/index.ts
import { Context } from 'aws-lambda'
import { config, loadParameters } from './config'

export const handler = async (event: any, context: Context) => {
  await loadParameters()
  return {
    value: config.values['/dev/key1'],
    success: true,
  }
}

This is our handler function that calls the loadParameters function to fetch our parameter and returns the value.

Let's have a look at config.ts where the main code for fetching the parameter and caching is included.

We'll start by defining the variables required:

// lambda-fns/config.ts

import { SSM } from 'aws-sdk'
import ms from 'ms'

type Config = {
  values: Record<string, string | undefined>
  expiryDate?: Date
}

const ssm = new SSM()
export let config: Config = { values: {} }

Here we have initialized an instance of SSM which we will use to fetch the parameter and the config variable that will be used to store our parameter. We can have multiple values fetched as well that can be stored here.

Notice the type of the config variable. It has two keys, the values which will store our parameters and the expiryDate that will invalidate our cache and fetch all the parameters again.

The expiry time is configurable and we will see how this is used in the loadParameters function next.

// lambda-fns/config.ts
export const loadParameters = async ({
  expiryTime: cacheDuration = '1h',
}: {
  expiryTime?: string
} = {}) => {
  if (!config.expiryDate) {
    config.expiryDate = new Date(Date.now() + ms(cacheDuration))
  }
  if (isConfigNotEmpty() && !hasCacheExpired()) return

  console.log('[Cost]: API called')
  config.values = {}
  const { Parameters = [] } = await ssm
    .getParametersByPath({
      Path: '/dev',
    })
    .promise()

  for (let param of Parameters) {
    if (param.Name) config.values[param.Name] = param.Value
  }
}

const hasCacheExpired = () =>
  config.expiryDate && new Date() > config.expiryDate

const isConfigNotEmpty = () => Object.keys(config.values).length

The final and most important function of our Lambda, the loadParameters function accepts a single value named expiryTime that is by default set to 1 hour and can be overridden. I have used the ms library to set human readable time periods which will be automatically converted to milliseconds.

We first check if the expiry date exists. If not, we set it to the value: current time + expiryTime.

Then we have this snippet:

if (isConfigNotEmpty() && !hasCacheExpired()) return

The following indicates that if we have values present in our config and if the cache hasn't expired then bail out of the function as we already have the values and we wouldn't want to call the API to fetch our parameters.

The above functions were just added to make the condition readable and they are defined as follows:

const hasCacheExpired = () =>
  config.expiryDate && new Date() > config.expiryDate

const isConfigNotEmpty = () => Object.keys(config.values).length

The next snippet is the entire fetching and setting of the parameters as we have already checked for the above conditions and we know that either the cache has expired or we do not have any values in our config.

console.log('[Cost]: API called')
config.values = {}
const { Parameters = [] } = await ssm
  .getParametersByPath({
    Path: '/dev',
  })
  .promise()

for (let param of Parameters) {
  if (param.Name) config.values[param.Name] = param.Value
}

In this, we simply added a log to know whether we're fetching the parameters and we have set the values to an empty object ({}). Then, we call the getParametersByPath method and pass the path dev we created in our resources file. Lastly, we loop over the parameters and add those to the values property in our config.

And we're done! To deploy this stack, run yarn cdk deploy or npm run cdk deploy.

Note: A prerequisite for this would be installing aws-cli and configuring the profile (I have configured the default profile in this case) with the Access and Secret keys.

Now, let's run this lambda by creating a test event from the console:

On clicking Test, we can see the logs and on the first run, we would see something like this:

We successfully get the result of the parameter we stored and if we check the logs, we can see API CALLED because the value wasn't present as it's the very first invocation and the call to Parameter Store was made to fetch the value.

Let's run the test event again and see the result.

We get the value, but notice that the log API CALLED isn't present, which means that the value was obtained from the cache :)

This values remains due the Lambda reusing the execution environment on subsequent invocations. This is although upto Lambda and we cannot configure this externally. Lambda can start on a clean state and we would need to add a condition to fetch the values again which we have done in the loadParameters function as seen above.

So that was it for caching values from the SSM Parameter Store in a Lambda function. Do not forget to delete this stack after you've done with it using yarn cdk destroy.

Thank you all for reading! Do like and share this post if you've enjoyed it :)