A cap of tea

A story about distributed systems, hype-driven design and the Socratic hardships of friendship

Fancy some tea?

• Me: Hey, fancy some tea?
• You: Sure, I'm always up for some tea!
• Me: Cool, let's share a cap of tea
• You: You mean a cup of tea right?
• Me: No no, I'm talking about a cap of tea, hear me out. If only there was a way for us to pick the right time to meet and share some tea, right? Wonder no more...
• You: [oh, no, not another pitch] ...

The pitch

"What is more powerful than the synergy of tea? Can we leverage the power of time to accelerate and optimize the engagement of the face time tea experience? Surely we can level up the tea experience to the next gen, to a rockstar, uber viral, vision.", I blurt.

"Wow, what a load of nonsense", you think, you feel a lot dumber just by having heard all that stuff. Those brain cells are not coming back.

An idea pops in to your head [Patent pending]: "The BS compressor". A lossy compression algorithm capable of boiling down all that nonsense into a simple "make tea better" or something. You start honing your time warping skills to see if you can start piping your live meetings through the BS compressor and gain precious hours of your life back.

You keep daydreaming of a better world with less fuzz and start thinking that maybe the problem with the world is that we are using the wrong font.

"Hey, are you with me?", I say while poking you in the arm.

"Sure, let's get this over with"

The product

"...as I was saying, we'll have this multi-tiered microservice architecture, where the service mesh communicates via an event bus ripe for enrichment..."

"Wait a minute, what does this product do?"

"What do you mean? The services, the bus, the replicas, the protocols, the consensus...", I trail off, confused. A small tear runs through my cheek.

"You mentioned tea?"

"Right tea. There's tea in there somewhere, but did I tell you about the orchestrator?"

You can feel the pressure building inside your head. The user-centered designer in you wants to scream in rage. After some effort you manage to compose yourself.

"But what about the user experience? What is this product for? Why would I use it?" you manage to say while swallowing sadness.

"Nah, don't worry about that. You know what's the best way to spice a depressing NPS, to cheer up a sad Kano (no, not the violent cyborg, the other one)? You're right, a beefy architecture diagram. An 8pt Courier New thing of beauty depicting in excruciating detail every little aspect of your architecture! Users love that stuff"

"Sure, whatever, what makes this idea so unique?"

"I'm glad you asked!"

"I bet you are"

The impossible trifecta

"Unlike previous unsuccessful attempts my idea captures the three fundamental properties of the best tea..."

"Color, aroma and flavor?"

"Wrong, you coffee-drinking muggle: consistency, availability and partition-tolerance"

"That's some powerful tea you're brewing" you say rolling your eyes.

"I sure am, let me tell you all about these wonderful properties" I reply completely immune to sarcasm.

"I can't wait"

Parsley and wood chippings

"When you've been in the tea industry for as long as I have, you know that that fancy green chai is expensive. No one in their right mind would serve that to customers.

Instead we just brew the chai with parsley or, in dire times, with wood chippings and green food coloring.

But what if two customers approach the counter at the same time and order a green chai? It'd be very bad for business if one of them would get the parsley chai while the other gets the wood chipping chai. That's just wrong"

"That's what's wrong?" you ask, making a mental note of never ever drinking my tea. "Anyway, I thought we were talking of a computer system, all that microservice stuff and all, but now it looks like you are opening a tea shop..."

"Pivot or die, my friend, pivot-or-die"

A guy named Chad

"How can you make sure every server knows which chai recipe to use at any given time?" you ask.

"Piece of cake, we just have a single server. That guy really loves his tea. He works 24/7 non-stop. When he's feeling drowsy he just takes one on the house. No one knows his name or where he came from so we just call him Chad.

Whenever we need to change the chai recipe we just tell Chad: hey Chad, we're running low on the good stuff, stop brewing parsley and switch to wood chippings. Sure thing, boss!"

Not knowing where to begin to express all the kinds of wrong here, you just opt to stay away from it and just ask: "So your solution to the consistency problem is having a single Chad?"

"Clearly, single Chad, zero fuzz"

Ouroboros' queue

"But wait, what happens if it's rush hour and lots of customers want their tea at the same time? Or, even worse if Chad collapses under the pressure." you are truly concerned now.

"Nah, that Chad has the immune system of a horse with a self-patching kernel. He can take it, but just in case, we are hiring a bunch of other servers, so that in case he's out or something we can still serve our customers"

Now that's more like it, multiple servers, some redundancy, higher resiliency to Chad nonsense, but you clearly see where things are going, so you ask: "But if you hire multiple servers how can you make sure they all follow the same chai recipe?"

An army of Chads

"Oh, you're gonna love this! We hired a bunch of servers. They all have colorful back stories, interesting personalities and unique network addresses, but to keep things simple I just call them all Chad.

Even better, they all have fancy Bluetooth ear pieces that keep them communicated at all times, so I can scream 'Chad, wood chipping time!' in the mic and all of them answer in unison: 'Sure thing, boss!', It's beautiful, I tell you"

"OK, let me get this straight: your approach to availabilty is hiring a bunch of random people, calling them all 'Chad' and relying on some Bluetooth dongles to transmit the stuff you bark over the microphone?"

"Right you are"

At this point you are tempted to ask "how do the Chads make sure all of them use the same recipe?" but you know me all too well to fall down this consensus rabbit hole.

So instead you ask your original question again: "But wait, you told me your consistency strategy was 'Single Chad, zero fuzz', but to have proper availability you now hired an 'Army of Chads', so what happens now?"

AC / DC

"Worry not, that was before, now that my Chads have their Bluetooth pieces the sky is the limit. Bluetooth is flawless, you know"

"Right, flawless...", you say with more eye rolling. Now that you think about it, your last expression of sarcasm seems like a lifetime ago. "Didn't you mention partition tolerance?"

Denial as an architectural pattern

"Yeah, partition tolerance, that is so overrated"

"Wait, what?!"

"To be honest, I don't believe in it"

"Hold on, you don't believe in network parti..." you trail off. At this point in our friendship you can spot a tangent from miles away so instead you try a more delicate approach.

"Let's say that a Chad runs out of juice on his Bluetooth dongle, or maybe walks in front of a microwave or something, what happens then?"

A rock and a hard place

"Hmm, let me think about it... well clearly we'd like to avoid that whole parsley/wood chipping debacle, so that Chad should stop serving tea until he can get his dongle in working condition..."

"Yeah, that looks like a very consistent approach, but what about all the angry customers queuing in front of your broken Chad?"

"Hmm, you're right, perhaps a better approach would be for that Chad to keep using the last known chai recipe, that way we can keep serving customers and everyone is happy..."

"Ah, the available approach, I like it. Although theoretically you could be serving two different types of chai at any given moment"

You can see my face in slow-mo as it dawns on me "so you're saying that because I went for that crappy Bluetooth stuff, that now I must decide between angry tea-less customers and the parsley/wood chipping debacle?"

The crux of CAP

Frankly you are surprised it took me so long to see the writing on the walls.

Network partitions are unavoidable whether because your Bluetooth ran out of battery, because the mailman was being chased by your neighbor's dog and dropped your letter, because you moved to too far from your wifi access point or because of a faulty network card.

And, when network partitions do occur, you are left with a tough choice:

• Drop availability in favor of consistency and partition tolerance (what's usually called a CP system) leaving you with a bunch of angry tea-less customers.
• Drop the strong consistency guarantee leaving you with a system that's both available and partition tolerant (i.e. an AP system) but serves all kinds of weird chai.

In the end you feel bad to see me so heartbroken so you give me some kind words of comfort:

"Hey, don't worry, it's not so bad, I'm sure 'A CAP of tea' will be a big success, you just need to replace that Bluetooth crap with something more reliable, like carrier pigeons"