Call external APIs with OAuth within Step Functions

Until last week, if you needed to call an external API from Step Functions execution, you had to use a Lambda function. And you needed to manage the responses and make the execution fail or success based on that. Also, most probably due to the fact that the external API is protected, custom code is required to manage the authorisation of the api call.

As you see this involves significant custom code which needs to be maintained by the developers.

Good news is, in the re:Invent 2023, AWS has introduced native integration to HTTPS APIs from Step Functions which allows to call to any 3rd party API endpoint as a step in the execution and, based on the response, you can perform any business logic remaining in the state machine.

How it works

In Step Functions, it utilises EventBridge connection to manage authentication credentials for the connection to the 3rd party API.

Create EventBridge Connection

In the EventBridge connection, you can select 3 different authentication options.

1. Basic (Username/Password)
2. OAuth Client Credentials
3. API Key

For this example, we will use OAuth client credentials as the authorisation mechanism.

To create a EventBridge connection with OAuth client credentials, you have to provide below information.

• Client id
• Client secret
• Auth endpoint 
• HTTP method

Once the EventBridge connection creation request is received, first it stores the provided OAuth credentials in AWS Secret Manager.

Next, it calls the auth endpoint with the given client id and client secret to obtain an auth token.

If the auth call is successful, then the auth token will be securely saved in the same secret previously created in the secret manager. And this token will be used in the calls to the api endpoint.

If the auth attempt failed, it simply de-register the EventBridge connection and it cannot no longer be usable.
Once the EventBridge connection is successfully created, we can use it in a State Machine as the authentication option for the api endpoint.

Create State Machine with 3rd party API

In the State Machine, simply add a "Call third-party API" step and, add the configurations - API endpoint, HTTP method. 

For authentication, enter the ARN of the EventBridge Connection created previously.

Also, you can configure the request payload that sends to the API and you may use the reference paths to build the payload from the runtime data.

Refreshing OAuth Token

Normally, OAuth uses short-lived tokens. When an API request uses a token that has already expired, it returns 401 Unauthorized error. In such a case, you will need to first renew the token by calling the auth endpoint and use the new token to call the API.

In the Step Function HTTP API call, this is taken care of by the EventBridge Connection.

Let's assume the API returns a 401 error which is the default OAuth response when the token is expired. In that case, EventBridge Connection will automatically call its auth endpoint and retrieve a new token. And update the secret manager entry.

So, in the State Machine, you need to set up "Retry" for the "Call third-party API" step for the specific error - "States.Http.StatusCode.401". This retry will automatically resolve the unthorized error without any additional steps.

You can always catch specific error using the HTTP error code as follows:
States.Http.StatusCode.[Status_Code]

In EventBridge Connection, OAuth tokens are refreshed when a 401 or 407 response is returned.

Since Step Functions execution uses EventBridge Connections and Secret Manager, necessary permissions must be set in the State Machine role.

Try this yourself

I have created a sample application that uses Step Function step to call an external API. Here, I have used CDK with Python. Github repository can be found at:
https://github.com/pubudusj/step-functions-https-api-integration

1. Clone the repository.
2. Install the dependencies and deploy the stack using CDK CLI.
3. Once deployed, it will create 2 Lambda functions with Function URLs.
4. One Function URL will be used as the API endpoint while the other will be used as the autorization endpoint.
5. EventBridge connection is also created with autorization endpoint Function URL.
6. And, a State Machine will be created with a single step to call the API endpoint and it has 2 retries set up if API returns status code 401. And authorization via the EventBridge Connection.

Testing

1. As soon as the stack is deployed, you can see there is a secret created in Secret Manager with the given client id, client secret and newly generated auth token.

2. You can initialize a Step Functions execution with below input.
   

   {"set401": false}
   

3. You can see it successfully completes and the output includes the return value from the API Lambda function. Also, in the Lambda logs, you can see the auth token from the secret is being used in the header as the Bearer token.

4. To simulate a 401 error, initialize a Step Functions execution with below input:
   

   {"set401": true}
   

5. With this input, the API Lambda function, will return a 401 error for the initial attempt. And the state will immediately start retrying. And the step will be successful in the retries.

6. If you check the secret in the Secret Manager, you will be able to see the token has updated with new value. Also, if you check the API Lambda's logs, you can see the request headers include different Bearer token values in the initial and the retry attempts.

Express vs Standard workflows

This feature works well in both Express and Standard State Machine types. However, it is wise to keep in mind that (even generally express workflows are cost effective compared to the standard workflows) cost of the express workflows depend on the execution duration too. 

So, if your APIs are slow, it will cost more. For Standard type, it will be a single state transition, so the time it takes to call the API will not be a concern for pricing.

Summary

The introduction of HTTP integration support in AWS Step Functions is a significant enhancement that simplifies the process of invoking external APIs. This feature not only reduces the custom Lambda code needed for API calls but also eliminates the necessity for custom code to handle complex retry logic when generating and renewing authentication tokens.

With this capability, developers can seamlessly utilise third-party APIs with minimal code, and enhance the overall efficiency of workflow orchestration in AWS Step Functions.

Useful Links

• Step Functions external endpoints AWS Blog: https://aws.amazon.com/blogs/aws/external-endpoints-and-testing-of-task-states-now-available-in-aws-step-functions/
• Step Functions external endpoints documentation: https://docs.aws.amazon.com/step-functions/latest/dg/connect-third-party-apis.html
• EventBridge Connection documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations.html#eb-api-destination-connection