This post is a digest of a conference talk I gave at the Nordic API Summit in Stockholm, Sweden. For those who prefer video to the written word, here's a link.

For a deeper dive, read on.

Building for Change

Much of the focus of this blog is creating services that can absorb change. Why? Because change is the most reliable business condition under which software must operate.

When we create systems that are amenable to change, it means we can experiment. We get to try lots of solutions to our problem. The cost of implementing any set of changes is decreased because our systems are not designed like Fabergé eggs.

New features and capabilities are easier to add without re-architecting the whole system or significant parts of it.

All of this comes down to reduced costs: a system that is less costly to change, to build, and to maintain. It is important to note however, that we speak not only of costs that can be strictly quantified.

We should also include reduced costs of debugging and reduced costs of our engineering organization's time, energy, and attention. Such cost reductions are absolutely worth taking a look at the pattern below.

State of the Art

If you design most API integration solutions like most companies, you are probably binding directly to the internal data model of a third-party service over which you exercise no control.

It looks like the following.

Directly binding to internal data models is most unwise.

This approach is by far the most intuitive design and is effective as far as it goes.

It is also brittle.

As soon as the services change at all, everything falls apart. The only way to resolve the broken system is a manual reconciliation—that is, a meeting between teams. Meanwhile precious time and, depending on the system, money is wasted.

This cycle repeats for as long as the services are in operation.

This is the state of the art in 2024.

Acceptance Criteria

How can we break the cycle? How can we create services that do not directly bind to the internals of peer services?

We will need two key ingredients: JSON Patch and self-describing messages.

Once we learn how to apply these ingredients, we will have everything we need to create services that exchange messages without needing to bind directly to the implementation details of peer services.

First things first: the core of the integration problem is the need for services to agree exactly on the details of message structure before they can communicate. This means as soon as the message structure diverges, the communication link is broken.

What would help is a way to translate one object structure to another on the fly. JSON Patch offers a solution.

JSON Patch

JSON Patch is a formal specification for applying a series of transformations to a candidate object to produce a result object.

Since a JSON Patch document is just a JSON object, it can be serialized. Because it can be serialized, it can be sent over the wire. Because it can be sent over the wire, we can create a link to it. But what do links have to do with anything?

To understand, we need to discuss self-describing messages.

Self-describing Messages

What is a self-describing message? It is a message that contains all the information required to process the message within the message itself.

If you're looking for examples, look no further than the webpage you're viewing right now. Everything required to render the page successfully is embedded in the HTML: from fonts to stylesheets to third-party scripts, images, video, etc.

When the page updates its CSS, you don't need to update your browser. No. You get a link to a stylesheet that contains the updated style code. Your browser is none the wiser. Another example: Have you ever received a package from FedEx, UPS, or DHL?

Packages are great examples of self-describing messages.

It probably looks something like the image above.

What is significant about it? If you have ever received a package—certainly any international parcel—you will notice all kinds of metadata (i.e. labels) on the box.

These labels tell processors how to process the package, which is the message in this example. Is the package fragile? Is it perishable? Does the package contain hazardous materials? Is the package accompanied by customs paperwork that offers hints about how to further process it?

What about tracking labels?

These are links to an outside system. They allow us to query a third-party data store for additional information about where the package has been and where it is going.

We can glean all of this information without needing to know what's inside the package. This is the essence of the self-describing message.

The Adapter Schema

The last thing we need is something called an Adapter Schema. This schema is one that neither of our collaborating services depend on for their business logic. This schema is only used to adapt one internal format to another while retaining all the fields of interest. Let's take a look at an example.

// Foo Service
{
  “name”: {
    “first”: “Jane”,
    “last”: “Doe”
  }
}

// Adapter Schema
{
  “firstName”: “Jane”,
  “lastName”: “Doe”
}

// Bar Service
{
  “customer”: {
    “given_name”: “Jane”,
    “surname”: “Doe”
  }
}

When we focus on the meaning of messages rather than their structure, we give ourselves the freedom to iterate on structure as needed.

Clearly, the above three objects refer to the same bit of information and that information has equivalent meaning. Structurally they differ but semantically they are the same.

The second object is our adapter schema and the first and third objects represent the data models of two distinct services. Our cooperating services will use this adapter to capture the meaningful data fields within the messages they exchange.

In the adapter schema we have flattened everything to the top level of the object but we needn't do so. The adapter schema can be as complex as our needs require. The key is that this schema is the only information that needs to be agreed upon ahead of time by our collaborating services.

A Solution Emerges

Once we have our adapter schema, we simply need to send a link to the patch document that will allow consumers of our messages to translate them to the adapter schema. From there, the consuming service can do any needed additional translations from the reliably stable adapter schema to the consuming service's own internal model.

Above, Foo Service can provide a link to a JSON Patch document that will allow Bar Service to translate from Foo’s internal model to the Adapter Schema. That patch document looks something like this:

[ 
  { "op": "move", "from": "/name/first", "path": "/firstName" }, 
  { "op": "move", "from": "/name/last", "path": "/lastName" }, 
  { "op": "remove", "path": "/name" } 
]

Recipe for transformation: a JSON Patch document is a set of instructions for moving from one object shape to another

As an alternative, services can exchange JSON Patch links that allow message consumers to translate directly to internal object models.

Regardless of approach, so long as the adapter schema is stable and each of our outgoing messages contains the link to the translating JSON Patch document, our services will not break owing to structural changes to our messages.

Message consumers are free to change their internal structures as much as they want without fear of breaking the integration. A reply message may also contain a link for a requesting service to translate back to the adapter schema before further processing of the reply.

In this way, all of our cooperating services can modify their internal object model as much as they like while still retaining the value of integration, the possibility of innovation, and most importantly, the ability to move at the speed of change.