A story about how specialisation could have ended in tragedy… and how to avoid it yourself
Has something like this ever happened to you?
You go on a website, and there’s a spinner which loads forever, or an empty screen that forces you to stare. There’s a disabled button that never enables, or a page that drains your battery only for the phone to die in the most inconvenient circumstances.
The software we use today doesn’t work; many of the causes boil down to specialisation.
One day, my colleague was refactoring the Big Ball of Mud inside of one of the many API services of their company. Initially, the members of his team were very defensive regarding his approach, so he paired with another colleague to craft a “proof of concept”. After he showcased the POC, the team was impressed. They liked the outcome and agreed that the changes should go to production.
The software of his company was serving thousands of users simultaneously around the clock. There were 3 kinds of UI front-ends: the iOS app, the Web app, and the Android app.
Everything went fine as the “DevOps Team” deployed the back-end change to production. However, minutes later, customers were unable to login to the iOS app. The Call Center was bombarded with a significant number of angry customers. Everybody was confused, but there were no doubts about what was the cause: the refactoring which had deployed was locking customers out.
So what caused the breakage?
The back-end API and the mobile front-end had an implicit contract. The back-end should return strings in a specific format so that the iOS app would interpret individual tokens from that format as variables. At runtime, clients would replace those variables with some data they had cached. The developers couldn’t observe the problem in any of the end-to-end tests, as the authentication mechanism for the test accounts didn’t require those tokens to be there.
There was no documentation for that functionality. The string format was only consumed by the iOS app, not by any of the other API clients.
One of the three front-end clients depended upon a specific string format of a specific server for their login mechanism to work.
The company had organised the teams by technology. Each of them was responsible for one client. There was the “iOS Team” owning the iOS app, the “Android Team” owning the Android app, the “Web Team” owning the Web app, and the “Back End Team” owning the API service. Many of the artefacts each of those teams produced had repeated business logic. That means changes in one application would require a developer to touch three to five different repositories to get the work done.
Thanks to the developer turnover rate in the department and the lack of documentation, the team who owned the relevant back-end service had no idea the iOS team was consuming such string format in production. The only way they found out they didn’t have control of their software was after refactoring a critical part of the service, deploying it to production and locking customers out of their accounts.
Product development worked like this: the feature started from the User Interface — web or mobile, and then each client had the responsibility to change as many services as they needed to get their work done. The company had multiple teams responsible for implementing similar UI functionality without clear boundaries of who owns what. Each developer was very comfortable with their stack, unwilling to discuss ideas with developers working with other technologies.
The communication cost for the development of a single feature was so high that each client ended up reinventing their own business rules within their apps. The back-end service ended up becoming a database over HTTP where every system had a direct dependency on it.
The effort to ship features in that company was significantly high; psychological safety was deficient. Everyone involved had all the incentives to produce software in big batches instead of incrementally.
It was a tragedy waiting to happen.
The segregation of people with “Front-End” skills from people with “Back-End” skills had a significant impact on the communication between teams.
Unless you looked from the outside, it was hard to see how the skill segregation was slowing down the software development process. Everybody was delivering their “part” of the job quickly and efficiently. The “other team” was always the problem.
The most straightforward solution to this is to consider the need for “The Architect” — that person sitting in the Ivory Tower dictating what should and should not be. Be careful! “The Architect” may be the right pill to treat the symptom, but it won’t cure the disease. It’s everyone’s job to understand the big picture. How do you expect to work with a purpose if you don’t understand what that overall purpose is?
“The Architect”, if you must, should be a facilitator rather than a dictator. They should mentor the other developers on how to spot the big picture to ask the right questions.
This story happened within a major financial institution that was too big to fail. It didn’t matter how slow they were: as long as the existing system continued producing millions of dollars of revenue, there was no incentive for them to change — technological stagnation at its best.
They were the lucky ones. Their software was generating enough revenue to outweigh all the costs for the design of new features.
However, your company probably won’t be as lucky as them.
Nowadays, it’s prevalent for developers to follow the technical path of major successful organisations. However, that doesn’t mean their path also works for you. There are many reasons why major tech companies have reached their state of success, many of them unrelated to technical excellence.
If the software works and the company is too big to fail, then it’s hard to push any change. However, if you’re not too big to fail, then you better act. Fast.
The idea of specialisation comes from the traditional mindset of a manual a production line. There are people positioned at different stages of the ramp, each responsible for constructing different parts of the product. Maybe that works for a production line, but it doesn’t work for an intellectually demanding task like software development.
Nowadays, specialisation seems to be the root of the most significant communication problems among software teams and their shift in culture. It doesn’t matter if you’re the best “iOS” developer or “React” developer in the world. If you can’t solve a problem that spans across multiple stacks, either by doing it yourself or pairing with those who know more than you in that area, then it’s harder to justify the value you’re bringing into the table. Pushing work over the wall to “the other team” doesn’t work, you have to own it to the end.
That doesn’t mean everyone should be the “Jack of all trades, master of none”.
Here’s the trick:
The best programmers I’ve met prioritise learning fundamentals and techniques, not technologies. They learn the trade-off of those techniques even before they understand the details of how to apply them using the tools they have. When they face a situation where one of those techniques makes sense, that’s when they invest the time to research and look up the details of how to apply them.
Learn fundamentals, not technologies. That's how you prevent becoming specialized in one thing while not having to understand everything.
It’s impossible to know everything; you optimise to learn what the options are so that you can dig deeper when necessary.
So what are you going to do?
Are you going to follow the trend of big companies and hire that great “React Developer” which is also a great “C# Developer” and “CS Guru”? Or are you going to work towards hiring that “Software Developer” who regardless of the programming language, understands the basics about architecture, software design, and other areas?
It’s time to look at software as an essential part of our lives and start to apply things that work, not to repeat the same thing over and over again, expecting different results. Specialisation creates silos; broad knowledge-sharing creates opportunities.
Don’t be like that tribe, waiting for the planes that’ll never come.
You’ll be waiting for a long, long time.
Thanks for reading. If you have some feedback, reach out to me on Twitter, Facebook or Github.
Top comments (0)