7 practices I learned while developing applications with API Gateway service integrations

Integrating API Gateway with other AWS services offers several benefits, including improved performance and reduced costs. Let's explore some practices I've adopted when developing application backends, with a focus on DynamoDB.

1. Background

I've been involved in the development of multiple web applications using both AWS CDK and other infrastructure frameworks. Additionally, I've created several applications on my own, ranging from small to large projects, where I had the freedom to choose the tech stack (typically TypeScript) and architecture (serverless when possible).

In this post, I'll share seven practices I learned while developing application backends.

2. Disclaimer

Your situation might differ from mine, and you might disagree with the practices I'm about to share, which is perfectly fine.

I prefer applying the single-table design principles in DynamoDB. You might not agree with this approach, and that's okay too.

The practices listed below are based on my experience. They have worked well for me, but that doesn't guarantee they will work for you.

3. Some practices for DynamoDB integration

You might already be familiar with many of the strategies below. Some I learned early on, while others I discovered more recently.

The examples provided illustrate direct integrations between API Gateway and DynamoDB. This means using VTL in API Gateway's request and/or response templates to define DynamoDB operations, and request and response payloads. The index keys (table and any local/global secondary index partition keys and sort keys) presented here were designed for a single table, which explains their unusual structure.

All examples are written in TypeScript, the language originally used to create CDK.

3.1. Use JSON.stringify() in CDK

Instead of writing template strings in the CDK code, we can use JSON.stringify() to create mapping templates.

For instance, if we want to request a specific user item and assume the username is johndoe, we might have a partition key PK and sort key SK like this:

{
  "PK": "USER#johndoe",
  "SK": "USER#johndoe"
}

The CDK integration code can be written as follows:

const integration = new apigateway.AwsIntegration({
  service: 'dynamodb',
  action: 'GetItem',
  // the GetItem action is actually a POST request to the DynamoDB API
  integrationHttpMethod: 'POST',
  options: {
    // API Gateway needs GetItem permission to the table
    credentialsRole: myRole,
    passthroughBehavior: apigateway.PassthroughBehavior.WHEN_NO_TEMPLATES,
    requestTemplates: {
      'application/json': JSON.stringify({
        TableName: tableName,
        Key: {
          PK: { S: `USER#$input.params('userName')` },
          SK: { S: `USER#$input.params('userName')` },
        },
      }),
    },
    // ...other properties
  }
})

The requestTemplates value is the same GetItemCommandInput object we would define in a Lambda function if we used one.

The $input.params('userName') expression extracts the userName query (or path) parameter's value, enabling us to generate dynamic user requests. For example, the request template above will map the /user?userName=johndoe query string to the item with PK and SK values of USER#johndoe in the DynamoDB table.

3.2. Use if/else to send a response if the requested item doesn't exist

Even if the requested item doesn't exist in DynamoDB, I want to ensure a response is sent back to API Gateway and the client.

For example, if the item with PK = USER#johndoe and SK = USER#johndoe doesn't exist in the database, the responseTemplates part of AwsIntegration might look like this:

const integration = new apigateway.AwsIntegration({
  action: 'GetItem',
  options: {
    integrationResponses: [
      {
        statusCode: '200',
        responseTemplates: {
          'application/json': `
#set($inputRoot = $input.path('$'))
#if($inputRoot.Item && !$inputRoot.Item.isEmpty())
  #set($user = $inputRoot.Item)
  {
    "status": "success",
    "data": {
      "userName": "$user.UserName.S",
    }
  }
#else
  #set($context.responseOverride.status = 404)
  {
    "status": "error",
    "error": {
      "message": "Item not found"
    }
  }
#end
          `
        }
      }
    ]
    // ...other properties
  },
  // ...other properties
})

The #if/#else/#end block works similarly to other programming languages.

If the item with the required PK and SK exists, we'll transform the DynamoDB response by removing the S type descriptor and adding the UserName to the userName property.

If it doesn't exist, DynamoDB will still return 200 OK to API Gateway because the request was valid and well-formatted. It's not DynamoDB's fault that a non-existing item was requested. Thus, we override the 200 status code from DynamoDB to 404 and add a descriptive message. (Some properties are omitted for brevity.)

Previously, I used Lambda functions to create these responses and error messages. Now, I try to avoid Lambdas whenever possible, opting for direct integrations with mapping templates. This approach eliminates cold starts and usually keeps response times under 100 ms.

3.3. Don't generalize templates - most are unique anyway

I used to spend a lot of time studying VTL syntax and trying to create a generic mapping template that would work for most requests and responses. However, I realized that API Gateway doesn't support the entire VTL ecosystem, and the supported VTL syntax is not well documented. This led to a lot of trial and error, consuming hours without significant progress.

Instead of creating a single, all-encompassing mapping template, I now create separate templates for each access pattern.

While this approach might seem contrary to clean code principles, it has proven more practical for me. If all table items only have string attributes, creating a generic template is feasible. I even have an example of such a template here.

However, as the application's access patterns grow more complex, most mapping templates end up being different. Handling multiple data types, lists (arrays), maps, and nested attributes in a single template proved slow and unproductive. Instead, I now do something like this:

#set($user = $inputRoot.Item)
{
  "userName": "$user.UserName.S",
  "address": "$user.Address.L",
  "age": "$user.Age.N",
}

I can also create nested response objects by iterating over lists and maps, and change key names to be different from the table attribute names (more on that below). This approach is quick, flexible, and easier to debug in case of errors.

I'm not against extracting repeated code into its own function. If I have a specific response or error format used in multiple templates, I write a simple function with a config argument to return customized messages while maintaining the same format. This is one reason why CDK is great!

3.4. Use specific data attributes

When designing an application with a single table, it's common to have multiple different entities, such as users, messages, events, and tasks. To avoid confusion, especially as the business grows and wants to identify individual entities with unique IDs, it's important to use specific data attributes.

I avoid using generic attributes like Id or Name because, with multiple entities in the table, it becomes difficult to remember which entity an Id refers to – is it a user or a task ID?

Instead, I use specific attributes, like UserId, UserName, TaskId and MessageId. This approach is straightforward and makes the code easier to read and work with.

3.5. Use ExpressionAttributeNames

DynamoDB has a list of reserved words such as Date (not case sensitive), which commonly appears as a database attribute.

Using reserved words directly in request input expressions will result in errors. For example, the following GetItem request template will not work:

requestTemplates: {
  'application/json': JSON.stringify({
    TableName: 'MyTable',
    Key: {
      PK: { S: `TASK#$input.params('taskId')` },
      SK: { S: `TASK#$input.params('taskId')` },
    },
    ProjectionExpression: 'TaskId, StartTime, EndTime, Date',
  }),
},

Here, ProjectionExpression includes TaskId, StartTime, EndTime and Date, which are attributes on the Task item we want to return to the client. We don't need the entire large item for this specific access pattern, only these properties.

To make the code work, we can use ExpressionAttributeNames:

requestTemplates: {
  'application/json': JSON.stringify({
    TableName: 'MyTable',
    Key: {
      PK: { S: `TASK#$input.params('taskId')` },
      SK: { S: `TASK#$input.params('taskId')` },
    },
    ProjectionExpression: '#taskid, #starttime, #endtime, #date',
    ExpressionAttributeNames: {
      '#taskid': 'TaskId',
      '#starttime': 'StartTime',
      '#endtime': 'EndTime',
      '#date': 'Date',
    },
  }),
},

Now, the template validator will accept it as valid code, and our stack will deploy. In this specific case, it's enough to create an expression attribute name for Date since it's the only reserved word among the projected attributes.

3.6. Payload and database attributes are the same

Keeping the payload and database attributes the same simplifies development.

Let's say we want to create a new Task with the following payload:

{
  "taskId": "3ecd0d50-6ece-468e-a6ed-66c58c9c6525",
  "startTime": "10:00",
  "endTime": "14:00",
  "date": "2024-07-08"
}

In this case, I want the Task item data attributes in the DynamoDB table to be taskId, startTime, endTime and date. Alternatively, I might make minor conversions, like taskId to TaskId, which is straightforward to implement.

When using Lambda functions for additional logic or input validation during create and update operations, I write a small function to convert payload keys to attributes. This conversion logic is always straightforward, and a generic function covers all use cases without exceptions.

3.7. Enable Amazon Q Developer

I found Amazon Q Developer (formerly known as CodeWhisperer) to be an excellent tool that helps me write code faster. It effectively predicts the code I'm about to write based on my previous code in the project folder. Q Developer can also recommend VTL mapping template code, and I'm satisfied with its accuracy.

I use VS Code as my code editor, and Q Developer seamlessly integrates with it. It also supports multiple programming languages, including TypeScript. Amazon Q works well in the command line, though this feature is currently available only for macOS.

4. Summary

Direct integrations between API Gateway and various AWS services can often eliminate the need for Lambda functions. Writing VTL request and response mapping templates can be daunting, especially for those with limited experience.
However, by applying some tricks and techniques, we can become more productive and create fast web applications more efficiently.

5. Further reading

Initialize REST API setup in API Gateway - API Gateway setup guide

Setting up REST API integrations - Relevant documentation section

Getting started with DynamoDB - DynamoDB basics and table creation

Integrations for REST APIs in API Gateway - The different integration types