The year is ending, 2019 is almost here and soon the 2010s will come to an end. This decade was marked by services from companies like Netflix, Google and Facebook thriving and becoming part of our daily lives.
For us in the tech field, these companies are examples. They build and operate systems that handle billions of users, 24/7, all around the world. Just think about it. That's a scale hard to grasp and many would doubt we could have so many people connected just a decade ago.
In this post I want to introduce a new concept came up with a couple days ago: R.A.D.I.C.A.L. systems, that is, systems that are Reactive, Automated, Distributed, Intelligent, Complex, Agile, and Lean.
I invented this acronym as a way to assemble these techniques that big companies are using to create successful systems that scale. I've been reading lots of books and watching lots of talks about these subjects and I realized there was no term to describe systems that mix all of them.
Most of the concepts here are not new, but they're becoming more and more popular thanks to the challenge of making systems -- R.A.D.I.C.A.L. systems -- capable of serving a huge number of people.
I'll start backwards by introducing each letter of the acronym in the reverse order. I believe this is the most constructive way to introduce the concepts. Whenever possible, I'll put links for those interested in reading more about the subjects I'm talking about.
This is my first public post about software engineering, so I hope you enjoy!
1. L stands for Lean Inception
Projecting systems from the scratch is hard, mainly when we don't know the business domain that we want to tackle. Lean Startup by Eric Ries taught us how to build, measure and learn to create a Minimum Viable Product (MVP) in an environment of uncertainty. On the other hand, Design Thinking brought to our attention the importance of research to have insights about problems in order to ideate and generate prototypes to get a specific solution. How can we mix these mindsets to create a new project?
Lean Inception by Paulo Caroli is a methodology that joins some techniques from Lean Startup and Design Thinking to start a new product. It does that by aligning the ideas of all stakeholders, product owners and developers involved in the project. The result (by the end of the one week session) is a set of MVP Canvas that describe the main goals, advantages and drawbacks of each iteration of development. They are accompanied by user stories that can be used by developers during the construction of the system.
A R.A.D.I.C.A.L system should use the Lean Inception to choose which and when features should be built.
2. A stands for Agile Development
The Agile Manifesto is 17 years old and being agile is no longer hipster but rather mainstream. It's hard to find a developer who does not agree with the core values of agility, where individuals and interactions, working software, customer collaboration and responding to change are valued respectively more than processes and tools, comprehensive documentation, contract negotiation and following a plan.
The flexibility of Agile made it the default mindset when companies start developing -- whichever techniques they choose to use, prepackaged in methodologies such as Extreme Programming (more focused on the developer-side), Scrum (more focused on the management side), Kanban (focused on the organization of tasks), and many others.
A R.A.D.I.C.A.L system should use the Agile Development to optimize how and where features should be built.
3. C stands for Complex Event Processing
Millions to billions of users produce many, many (did I say many?) data, all the time, and we need a way to handle it. In an era of Internet of Things where more and more devices are connected together, we need a way to handle all the signals they produce when interacting with humans and other machines.
Complex Event Processing is a theory that advances many (big) data processing techniques to establish a well-defined method to deal with infinite data sets, generating ways to query, transform and mix different streams. It gives a framework to abstract dataflows, processes and exceptions that are happening inside a system.
A R.A.D.I.C.A.L system should use Complex Event Processing techniques to track and analyze the data inside it.
4. I stands for Intelligent Data Transformation
Artificial Intelligence and, in particular, Machine Learning are trending in the last couple years since humans can no longer grasp the amount of data we are producing. Pattern Recognition, Computer Vision, Natural Language Processing are some of the areas that AI/ML are making the most impact.
The result is a new generation of systems that make things that we could only dream of a decade ago, such as voice assistants, autonomous cars and reusable rockets. As the ecosystem matures, using algorithms and models from these areas to find out valuable information to your product is no longer a task only for PhDs.
A R.A.D.I.C.A.L system should use Intelligent Data Transformation techniques to learn and generate insights from the data inside it.
5. D stands for Distributed Environments
Some say that Cloud Computing is making to computation what the first power plants made to electricity. We no longer need to lose time maintaining our own servers to run and scale a system for new users. The cloud, however, brought many new challenges for developers. We no longer have the control of when a machine will be available, neither where exactly our systems are running.
Distributed systems are as old as the first networks. It is a cheap solution to increase processing power and the cloud made it much cheaper. However, the CAP theorem made the words availability and consistency known by every developer that deals with these systems. This mix of theoretical and practical challenges pushed the creation of a new generation of tools that encapsulate the best patterns to manage these environments.
A R.A.D.I.C.A.L system should use Distributed Environments to be scalable in a simple and unrestrained manner.
6. A stands for Automated Infrastructure
If servers are out of our hands (thanks to the cloud), automation is very important to guarantee that we can run our services anywhere (particularly with the least vendor lock-in as possible). Infrastructure as Code was the first step to enable developers to take part of operators responsibility, leading to the much broader Cloud-Native movement (which includes technologies for virtualization, orchestration, observability and many other buzzwords) and the Serverless movement (which tries to free developers to think in nothing but their business logic).
At the same time, DevOps is shining as a mindset to close the gap between development (creating new things) and operations (maintaining existing things), allowing the full ownership of products by those who created them. This heavily relies on and allows Continuous Delivery of new features, where they can be packaged, tested and deployed with shorter cycles.
A R.A.D.I.C.A.L system should use Automated Infrastructure to be deployable in a safe and reproducible manner.
7. R stands for Reactive Microservices Architecture
Complex business cases may demand large systems that span many business subdomains. Domain-Driven Design by Eric Evans brought the concept of a ubiquitous language that should be used to describe these systems and bounded contexts that divide them. These are the building blocks for a Microservices architecture, that explores their semi-independence to create a system composed by many independent services which can be developed by standalone teams.
But building microservices is not a silver bullet for modern systems since there is a set of prerequisites in order to adopt them instead of a monolith. Anyway, the architecture still can become a big ball of mud. The Reactive Manifesto indicates how creating responsive, resilient, elastic and message-driven services may help to strength systems for scale.
A R.A.D.I.C.A.L system should use Reactive Microservices Architecture to encode complex business models and domain-specific knowledge.
That's it! What do you think about R.A.D.I.C.A.L systems? I'm sure they are not easy to make, and that grasping every concept requires lots of learning -- but it's been a very nice journey.
I want to thank my friends Fernando, Rafael and Victor for the valuable feedback. This post is just scratching the surface, but I want to highlight that there are systems -- R.A.D.I.C.A.L. systems -- that mix all these concepts.
If you want to discuss more about these ideas, please reach me at Twitter or LinkedIn.
Top comments (8)
Interesting article though I am curious why you chose to go through the list L-to-R rather than R-to-L, it caught me off guard while I was reading it given the acronym you've coined.
Your article did help me realise some terms for things I was already looking into like CEP so thanks for that! :)
Hi James, thanks for your comment!
I think this approach is more constructive, that is, it reminds me of the steps I would think about when building a system with these techniques:
Lean inceptionhelps to understand what to build.Agile methodologiesexplain how to build it.CEPdeals with how to process the data.AI/MLhandles how to learn from the data.Distributed environmentsare about where to run the system.Automated infrastructureis about how to run the system.Reactive Microservices Architectureputs everything together in a single structure.In my mind it's more natural to explain
1 → 2,3 → 4, and5 → 6, although I feel I could begin with any of the pairs1-2,3-4, and5-6since they talk about planning, data and infrastructure respectively. I also like ending with the architecture because it is really what defines the system. In the end this coincided with the acronym backwards :)I'm glad you learned some interesting new terms! Take a look at the e-book Fast Data Architectures for Streaming Applications by Dean Wampler, it's one of my inspirations for this section about big data and complex event processing.
Yeah, I see what you mean when looking at it like that - each point builds upon the next. It is just a shame that order isn't the same order as the actual letters in the acronym - L.A.C.I.D.A.R just doesn't have the same ring to it.
Hi Aleksei, thanks for your comment!
You are right: thare are many scalable systems that don't need to use everything I described. As you said in your other comment, there are no silver bullets. Using so many techniques together brings all their advantages and their drawbacks together.
Now about the AI vs. ML discussion, I often find people describing ML as a subset of AI, maybe better described by the name of "statistical learning". In my understanding, AI is a wider field that uses different mathematical frameworks (another example, first order logic and graph theory) to create automated decision making. What are your thoughts?
Here in Brazil we have a quite large fintech ecosystem where more companies, in particular banks, are applying machine learning for risk evaluation, fraud detection, etc. Other interesting use cases I saw in other domains was the use of ML in the infrastructure level to optimize resource usage.
Surely, as you said, many systems don't need to apply everything to be highly scalable (bank systems running in mainframes or chatting systems built with Erlang are nice examples). However I wanted to highlight some systems use all those concepts together and we could give a name to them.
This all seems fairly substantive. I'd caution against making it too buzzwordy, however this might play well in corporate environments.
I approve on the idea.
Hi Ben, thanks for your comment!
The text is indeed full of buzzwords. I definitively don't want to promote using them indiscriminately. Applying those techniques together brings all their advantages but also all their drawbacks. And it's by no means easy to do -- maybe something we may target to evolve our system if it ever needs to handle such a large scale.
Nevertheless, I want a way to aggregate interesting concepts to highlight that sometimes they are used together. I really enjoy to try to take this cross-cutting look at system building and that's when the idea of the acronym appeared.
It's a good list just that you miss out on the human aspects of it.
You can read books with psychological or ethics to understand human problems like "Hooked" by Nira Eyral and Ryan Hoover so that we don't build stuff that is against ethical issues like Target focusing on Pregnant women or Cambridge Analytical and recent court hearing of Google.
Hi Max, thanks for your comments!
I really focused more on the technical side of those systems, the only place where I touch the human aspect was where I listed the agile methodologies -- whose primary concern is not related to these social / ethical / diversity issues you talked about.
I'll take a look on your reading recommendations, thanks!