Embrace Eventual Consistency

Soon or later you are going to deal with eventual consistency as your system scales.
You can't run from the CAP theorem, you can't have these three: Consistency, Availability, and Partition Tolerance.

High Availability

To create a high availability service you need redundancy.
If you are running a single pod for a service in your Kubernetes, when it goes down your service goes down. To create a high availability in this scenario you always need to run at least two pods, if one goes down you still have the other until the other one restarts.

You have the same situation for databases. If you run a single node for your database, and it goes down, all applications that depend on it will go down.
One approach is to create a replica set that will have one primary node that receives the writes and other nodes secondary that can be used for reading. When the primary goes down, another node will be elected primary and your users won't notice.

CAP Theorem

In database theory, you can only choose two of these three options in any distributed data store:

• Consistency: Every read receives the most recent write or an error
• Availability: Every request received by a non-failing node in the system must result in a response
• Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.

A distributed data store is any "database" that is stored in more than one node.

No system can prevent a network failure causing a network partition.
When this happens you need to choose between consistency or availability.
Without a network, you can't ensure your system is consistent and has the most recent write, so you need to decrease availability to ensure consistency.
Another approach is to choose availability and proceed with accepting requests but risk consistency.

Not everything needs to be strong consistency in your system. Eventual consistency can be enough for many use cases.

Read followed by Write

Some patterns should be avoided when embracing eventual consistency, like trying to read after doing a write. To solve this you can use the value that you already wrote, or you should read from the primary database to avoid eventual consistency in this particular case.
Every time you read from the primary, you reduce the read scalability of your services as you are consuming more of the primary than the secondary.

External APIs/Webhooks

Some external APIs/Webhooks could show the eventual consistency behavior, as they propagate the changes to replica databases or other microservices.
One pattern to solve this is to delay the API call after some time, read more about it at Handling Eventual Consistency in Webhook

In Resume

Understanding these concepts can help you model better the architecture of your software systems with the correct trade-offs that you need for each part of it.
Eventual Consistency, High Availability, CAP Theorem, Availability, and Consistency are topics that you need to understand to design data-intensive applications.
My advice is to embrace, learn, and understand them.

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Imbattable, by Pascal Jousselin http://pjousselin.free.fr