Handle optional input parameters in AWS Step Functions without Lambda

TL;DR

If you want to apply data processing to an input object in your state machine without the well known struggle of dealing with optional parameters, you can use the HandleOptionalParameters CDK construct as follows:

const transformInputValuesTo1ItemArraysTask = new HandleOptionalParameters(
  scope,
  "Handle optional parameters",
  {
      requiredProperties: ["requiredProperty"],
      optionalProperties: ["optionalProperty1", "optionalProperty2"],
      // Apply States.Array intrinsinc function on every input values:
      dataProcessing: "States.Array($.value)",
      // Other examples:

      // Encode every input values and return the object with same input keys but with encoded values:
      // dataProcessing: "States.Base64Encode($.value)",

      // Concatenate the value with the property name, with the JsonPath $.propertyName:
      // dataProcessing: "States.Format('Current key is {} and associated value is {}.', $.propertyName, $.value)",
  }
);

new StateMachine(
  scope,
  "State machine using HandleOptionalParameters construct",
  {
      definition: processObjectValuesTask,
  }
);

🔴 Context: AWS Step Functions throws if the provided JsonPath is not present

AWS Step Functions lets you do data processing on a state machine execution input thanks to its intrinsic functions.

For example, let's say you need to put every values of an object input in an array:

To transform a value to an 1-item array containing this value with Amazon State Language, you can use the States.Array() intrinsic function:

To transform every values of an input object, you can apply States.Array() on every properties:

However, when dealing with object inputs, you have to make sure that each property you want to process is present in the input, otherwise the execution will fail ❌:

An error occurred while executing the state 'Transform input values to array' (entered at the event id #2). The function 'States.Array($.optionalProperty)' had the following error: The JsonPath argument for the field '$.optionalProperty' could not be found in the input { "requiredProperty": 1 }

Indeed, if foo is actually undefined:

You thus need to handle these optional input parameters.

🟠 Choices: the false good idea for large inputs

Choice is another common flow state of AWS Step Functions that enables you to decide on which branch to continue the state machine execution, based on a condition evaluated at run time. You can thus use a Choice state to evaluate the optional property's presence in the input, and apply the States.Array() function only if the property is defined in the input:

However, if the input has many optional values, your step function may look like this 😓:

This causes scalability issues:

• 🧑‍💻 DX: as you can see in the execution graph view, debugging is painful in the AWS console
• 🛑 AWS limitations: in my case, Cloudformation template hard size limit of 1MB is reached with 10 optional properties
• 💸 Cost: in a standard workflow, billing depends on the number of state transition which is proportional to the number of optional properties
• 🚀 Performance: Choices are not parallelised which leads speed to decrease with the number of optional properties

You thus need to handle the scalability of your optional parameters.

🟢 Let the HandleOptionalParameters CDK construct handle the optional parameters for you

The HandleOptionalParameters CDK construct processes every defined values with whatever data processing you provide, while ignoring the undefined ones. And it does it in 9 common flow tasks, regardless of the number of optional parameters 🎉:

Let's break it down:

Conclusion

From now on, stop using Lambda for handling large state machine inputs with many optional parameters ! 😉