Netcore Igniting plug-ins with AWS Lambda and Event Bridge

Authored by: https://www.linkedin.com/in/swayamprakashbehera/ and https://www.linkedin.com/in/ivikramsahu/

About Netcore Cloud

Netcore Cloud is a full-stack marketing technology platform with 20 years of comprehensive martech experience. It supports the brands with several great features, like

1. Client interaction and experience
2. Platform for Consumer Data
3. Personalization.
4. Email API
5. Planning the journey.

Thus far, Netcore has helped almost 6500 brands in 18 countries get more international customers. The whole product line and roadmap for Netcore are built on these three pillars:

• Customer-centric.
• Creative.
• Scalable on a global scale.

Now that you are aware of Netcore, let's talk about how we were able to fix our plug-in development model, which ultimately turned out to be a best-in-class architecture for us in terms of development, dependability, and scalability.

🔒 The Problem : Multiple channel formation, which slows down delivery.

Netcore, the top worldwide Martech platform, is required to make a number of alterations in order to abide by local laws and regulations. We must respond to customers' needs more swiftly because consumers' preferred channels vary by country.

As we expand globally, integrating channels becomes increasingly important for Netcore. To meet the growing demand for channel integration, we are investing time and resources in developing more event integration channels.

To expedite delivery and save on development time, we have opted to create a unified architecture that can address the challenges of channel development. This way, we can avoid having to build or develop a new integration every time we need to connect with a new remote channel.

We are focusing on a "create it once, use it forever" approach to sustain our efforts and improve our architecture. This approach will enable us to provide faster and higher quality results.

Let's review the specific steps we took to automate the channel creation process using AWS Lambda and Event Bridge.

🔑 The Solution : Consistent Design for Continuous Delivery

Before moving on, let's familiarise ourselves with a few terms we'll be using regularly,

Serverless : A serverless architecture is a way to build and run applications and services without having to manage infrastructure. Your application still runs on servers, but all the server management is done by AWS.
You no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems.

AWS lambda : AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.

EventBridge : EventBridge is a serverless service that uses events to connect application components together, making it easier for you to build scalable event-driven applications. Use it to route events from sources such as home-grown applications, AWS services, and third-party software to consumer applications across your organization.
Event Bridge provides a simple and consistent way to ingest, filter, transform, and deliver events, so you can build new applications quickly.

Solution Overview

Each channel is designed to perform a specific task triggered by an event. Therefore, let's explore the various events that should be considered when attempting to address this issue.

In general, let's consider there are two different events:

1. Events that require immediate action (or aggressive SLA).
2. Events that can be processed later with an affordable latency (1–2 minutes).

In this example, category 1 events with strict SLAs are directed to the Fast Bus, which is an EventBridge Custom Bus. Meanwhile, category 2 events are sent to the Slow Bus, which is another EventBridge Custom Bus. Both types of events originate from the "Marketplace App Action" module of the Journey Orchestration engine, which is built on Apache Storm (more on this in an upcoming article).

Regardless of the event category, each event must meet specific data validation and transformation requirements that are unique to the channel through which it will be processed.

Theory of Input transformation

To process the inputs sent to the Fast Bus and Slow Bus based on their classification, the appropriate EventBridge Input Transformation is necessary.

For Fast Bus, After the transformation, the process calls a specific Amazon Lambda function, known as the "Function Leader," to determine the next Lambda function that needs to be invoked. This group of Lambda functions is referred to as the "Function Followers," and they are invoked by the "Function Leader" Lambda.

For Slow Bus, Upon transformation, events are sent to Kinesis Data Firehose. Kinesis Data Firehose is a cost-effective way to store events in an Amazon S3 bucket. It divides records based on the data found in the record or computed using a Lambda function. This partitioning information is utilized by Kinesis Data Firehose to generate keys in the bucket and store relevant records inside those keys.

When events are sent to S3, another Amazon Lambda function is triggered. This function retrieves 1000 records at a time from S3 files and generates a single batched event to send to AWS EventBridge. By batching 1000 records into a single event, we can reduce the cost of EventBridge.

After the event has been processed through either the Fast Bus or Slow Bus, it is published to the "Curated Data Pool" of the AWS EventBridge Custom Bus. The data is presented in a JSON format that follows a single, universal structure independent of any specific channel or event type.

An Amazon EventBridge Rule is utilized to filter events that need to be sent to a specific remote destination. The input JSON is transformed using input transformation into a format that can be understood by the remote API Target.

The EventBridge API Destination is utilized to configure connectivity to remote endpoints.

The architecture makes it possible for us to onboard integration to any remote platform in the future with:

• Only a number of configurations
• There is no extra code
• Indefinite scales
• Speedier delivery
• Superior quality

Architecture Overview

The architectural diagram can aid in understanding the various system components and their interactions. By utilizing the diagrams, data flow, and annotations shown below, one can easily comprehend the function and purpose of the system.

The Data Flow

Note : Please refer to the above architecture diagram when reading the data flow.

1. Events are generated from the "Journeys" module, with Apache Storm directly publishing events to EventBridge.
   • Event with immediate action are published to Fast Bus (1a)
   • Event with affordable latency are published to Slow bus (1b)
2. The EventBridge Input Transformation is utilized for data transformation.
   • The lambda responsible for preparing the data for the Amazon Function Leader lambda is triggered by transformed Fast Bus events. (2a)
   • Transformed Slow Bus events transfer data to Kinesis Data Firehose using the partitioning key. (Tenant ID) (1b to 2b)
   • Kinesis publishes batches of data to an S3 bucket, and when an S3 put event occurs, a lambda function is triggered to prepare the data for an AWS Function Leader lambda. (2b to 2c to 2d)
3. A Lambda function selects 1000 records from an S3 file and creates a single event, which is then published to AWS EventBridge. (3)
4. Amazon EventBridge triggers the Function Leader Lambda to orchestrate flows. (4)
   • Having a table for orchestration in Amazon RDS MySQL is a good way to keep track of the processes that need to be followed based on the destination channel of an event. This table can include information such as the next lambda function in the "Function Followers" group that should be invoked by the function leader lambda.
   • The Function Leader Lambda uses this table to determine which Function Followers Lambda to invoke for each event based on its destination and current state of the process. The Function Followers Lambda is responsible for sending the event to the correct remote endpoint by asynchronously invoking the lambda function.
   • By using an orchestration table, we can easily add new channels and destinations without needing to modify the code of the Lambda functions. Instead, we can simply update the table with the new information, and the Function Leader Lambda will automatically route the events to the correct Function Followers Lambda based on the updated table.
5. The Function Leader selects which lambda of the Function Follower to call based on the step count (1, 2, 3, and FINISH). While sending the vent to function follower, it passes on the "step" value. (5)
6. Each function follower Lambda increases the step number and sends the processed event to the Function Leader Lambda. The Function Leader Lambda then evaluates the next step based on the channel-specific orchestration table in Amazon RDS MySQL and triggers the appropriate function follower Lambda for the next step. This process continues until the event is fully processed and ready for delivery to the remote destination. (6)
7. Once the event has gone through all the necessary processing stages, the Function Leader Lambda will submit the event to the AWS EventBridge Custom Bus, where it will be available for further processing or routing to external systems via API Destinations. (Curated Data Pool). (7)
8. The EventBridge Rule is used to filter events that need to be sent to a specific remote destination, and then the input JSON is transformed using input transformation into a format that the remote API Target can understand. Finally, the EventBridge API Destination is used to configure connectivity to remote endpoints and to send the event data to the designated destination. (8)

Why AWS lamda & Event Bridge?

AWS Lambda and Amazon EventBridge are popular serverless computing services by Amazon Web Services. It becomes great choices for building modern, scalable, and cost-effective serverless applications, especially when integrating with other AWS services.

We had some AWS services already running that were helpful for solving our business needs and met our technical requirements.
Netcore Team has taken a significant step towards achieving their platform's goal of establishing a "No code scalable" platform.

Conclusion

This article explained how NetcoreCloud's platform achieved independence, scalability, and speed by leveraging a combination of Amazon Lambda, EventBridge, Kinesis Data Firehose and S3. The architectural overview of the system was discussed in detail.

After improving our main foundation, we are aiming to be more flexible to add more communication channels.

With lambda's grace and speed, we have scaled our clients' campaigns at the busiest periods and committed to deliver new channels at a blazing fast speed.