What production environments can learn from McDonalds

I figured it would be apt to have my first Dev.to post be about something I find in common between my current development job and my entry-level job.

Imagine this: You walk into a McDonalds to get lunch. Among the usual menu items (and the occasional waiting around for the menu to come back because an ad took over the entire screen), you see a new food item that wasn't in the menu the last time you went there.

After ordering it out of interest, the manager asks you how you liked the new item - or just surreptitiously look at you to see how you would react on your first few bites.

There are three possible outcomes for this scenario:

1. You hate it. You take one bite, not finish your meal, and let the manager know that you didn't like this new food item. Nearly all others hate it as well, and the following week, you don't see the new food item anymore.
2. You love it, as well as a lot of other people. This item eventually becomes a permanent menu item on all McDonalds restaurants nationwide, possibly even replacing something that wasn't selling as much as this new item.
3. You and probably half of the store's customers like it. Since not everybody hates this new menu item, it makes it to the menu across all McDonalds restaurants as an "a-la-carte" item: it's not popular enough to be have its own number on the Value Meal list.

This is a very common occurrence that happens at most major fastfood places. When I worked at McDonalds, we called it an LTO - a "limited time offer" - and is usually only rolled out to one region to see how customers would like it before making a decision to feature it nationwide.

This concept is not something new in the restaurant industry, and it also has a place in software development. It's called canary deployment, and involves:

"pushing programming code changes to a small number of end users who have not volunteered to test anything".

"Deploying code to non-volunteers?! What a dubious production model!"

The concept might sound suspicious (why would you push code to non-testers?), but the reality is that canary deployment only seems dubious because source code is unlike something tangible like a burger, or a milkshake.

In fact, canary deployment is something that is already being implemented in most enterprise software. GitLab has procedures on how to enable canary deployment and describes it as a "popular continuous deployment strategy". Klarna, a company that provides financial services to online businesses, suggests in their dev blogs that canary deployment adds a huge level of confidence to software releases. The Netflix Tech Blog article "Confidence in the Canary" shows that canary deployment takes up a big part of their testing pipeline, right in the middle between regression tests and red-black tests:

Canary deployment is as common as Shamrock Shakes, Chestnut Praline Lattés, and all the other delicious things that we didn't really ask for to be added on the menu, but we still nonetheless eat. After all, isn't the reason we keep coming back to McDonalds is to see what exciting new burger we can try?