SRE is becoming a very popular term in the DevOps and generally the software development world. Probably some of you have already heard about it, but are not sure what it is exactly.

So this article gives a detailed look at what SRE or Site Reliability Engineering really is with the goal to clarify all questions and doubts around it 🙌

This is the written version of my new youtube video ✍️ 🙂

Table Of Contents 📝

• Why was there a need for SRE?
• What is SRE? - Official Definition
• What is system reliability and why it's important?
• How to make systems reliable?
• SRE in Practice: SLA & Error Budget
• SRE Tasks and Responsibilities
• Who is doing SRE? SRE Role
• SRE vs DevOps

First of all, I recommend that you read my recent DevOps article first, where I explain DevOps in detail. This will make it definitely easier to understand the topics in this video 😊

How SRE emerged and why was there even a need for SRE? 🤔

In a traditional software development process, we have developers and operations as two separate teams. Each of them with their own goal:

• Developers: Want to push out application changes as fast as possible to the end users
• Operations: Want to keep the application stable

So Operations are very careful about each and every change and this causes a conflict of interest between these two roles, forcing them to work against each other instead of collaborate:

And DevOps was actually introduced to help fix exactly this issue. However, while DevOps made release process faster, these releases were not as stable as ideally wished by DevOps principles. Plus in DevOps team there was no dedicated role or person that actually focused full time on keeping systems reliable.

And that's how the need for SRE and a Site Reliability Engineer as a separate role emerged. 💡

So what is SRE?

What is SRE? - Official Definition

SRE was conceptualized at Google by Ben Traynor, a software engineer, who was given a task to run a small team of other software engineers to do what used to be Operations work.

And according to his own definition:

SRE is what happens when you treat operations as a software problem and staff it with a bunch of software engineers. 👩‍💻🧑🏻‍💻

At its core:

But this definition is of course too vague and high level to really understand how it's implemented in practice. 🧐 So let's break it down and analyze each part of this definition step by step. 👍

What is system reliability and why it's important? 🧐

First of all, what is a system that we want to keep reliable? Or what does a system even mean in this definition?

What is a System?

The system is the servers, infrastructure, the platform. So the whole deployment environment, where the application runs.

What is Reliablity?

Now what exactly is reliability and why is it so important to keep our systems reliable? 🤔

Unreliable Services ⛔️
Imagine you work with emails daily and your email provider is down once a week or your online banking application is down and not accessible regularly. This would be unreliable services. 🤨 You can't rely that it's available when you need it.

Reliable Services ❇️
On the other hand, many popular services like Gmail, Twitter, Youtube etc are rarely inaccessible. 👏 So these systems are pretty reliable.

But the thing is, users usually do not notice reliability of the system. 🙄 It only becomes visible when something goes wrong and the services are down. Do you remember the recent outage of Facebook, Instagram and other related services that made huge news?

What about AWS server outages that also affected other applications that were hosted on AWS?

Of course everybody noticed and knew about it when it happened. So the more popular and bigger the product or service and the more used, the more impact it will have 👀, if the service had an outage, which means more their team should worry about its reliability.

Why reliability is important?

What are the effects or impact of outages or system unreliability? For most of the services this is a lot of unhappy customers and lots of lost revenue:

Like imagine an online shop is down on a holiday or an online bank is not working, because of traffic overload. This means lots of lost business, because people cannot order anything on that shop.

How to make systems reliable?

Okay, so we understood that systems need to be reliable, but how do we make a system reliable or asked differently:
What makes a system unreliable, what affects its reliability? 🤔

The main cause of system becoming unreliable is, when you make changes to your system ⚡️:

Like change something:
⚡️ in the infrastructure
⚡️ the platform, where the application is running
⚡️ the application itself and its services
⚡️ and so on

These changes may cause a disruption and break something in the whole setup.

Bad Solution 👎🏼

As a solution we can say no changes allowed or limit the amount of changes 🙅🏻‍♂️ to keep systems reliable, but that really limits the business. We want to make changes and improvements to our application to make it better and increase its business value and stay competitive etc. 💪

Because if our competitor is bringing out new features, we need to keep up, and that's the main focus of software developers, to make those changes and improvements.

But on the other hand, if the application is not accessible that's also bad for business, because you may have awesome features, but nobody can use it. And it's the operations job to take care of that and make sure the application is accessible.

This means Devs want to release fast and Ops want to keep stability. So traditionally, Devs would make a change and Ops would analyze with hundreds of checklists and mechanisms to make sure the change would not affect the system:

This whole analysis and evaluations slows down the release process and that's been the major challenge of the traditional way of software development. And that's exactly what DevOps and SRE try to solve. 🚀

So what's the specific solution of SRE here? 💡

Well, SRE tries to automate the process of analyzing and evaluating the effects the change will have on our system reliability. Automation means no checklists or discussions of operations team, whether to release the change or not or what threats and risks are involved:

Instead, the evaluation is based on automated processes and this makes releasing changes fast and safe at the same time. 🚀

SRE in Practice: SLA & Error Budget

Now how is that automated evaluation done? 🤩

The way it works is using what's called SLAs. So what is an SLA? SLA is basically, how reliable a system is going to be to its end users. So how often it's going to be up and how often it's going to be down. And it's expressed in percentage.
A service that works all the time, is never down, has a hundred percent SLA. 💯

No need for 100% Reliability

Now you may be thinking: "Of course any service should be 100 reliable right isn't that a natural goal?" 👀
Well, not really. First of all, it's very hard to achieve 100% reliability and there are very few services in the world that actually need a 100% SLA.

So for example, if your internet provider or the customer's device itself is not 100% reliable, which is the case with most laptops, mobile phones and so on, then your service does not need to be either. It can be available maximum at the same rate as the underlying network or device. 💡

And in those cases a reliability of three nines or four or five nines may be enough, so that users don't even notice that there is an issue. 👌

The closer, you try to get to 100%, the more effort it is, which as you see is an unneeded effort, because you don't need 100% SLA for most applications

Examples of SLA

For example you can define a service level agreement about the accessibility of your application:

Source: https://en.wikipedia.org/wiki/High_availability

So for example a 99 SLA for application accessibility would mean that the system can be down maximum 3.65 days in a year. An SLA with 5 nines or 99.999% allows an application to be unaccessible maximum 5 minutes a year, so the rest of the time it should work.

You can actually define multiple such agreements or SLAs, not just the accessibility or availability of the system.

Other SLA examples:

• application response time
• error rate

For example if you have an application that serves a million requests a week with 99% SLA, you define that 990 000 of those requests will be successful.

Who defines these SLAs? 💁🏻‍♀️

Okay now you may be wondering who defines these SLAs? So basically who decides how many requests must be successful out of these million requests or how much downtime is allowed for the application? Who makes this kind of decisions?

As this decision affects the end users and their user experience, naturally business people are also involved in this process. So business people together with the engineers like SRE and DevOps Engineers decide together what service level agreements they want to define for their application:

Based on the industry benchmarks, competition, user feedback and so on, business people will define the desired SLAs on a higher level and the engineers will then define them on a technical level and also make sure to integrate them into their DevOps and SRE processes.

Error Budget

As I mentioned SLA for availability defines how long the service should be available or how much downtime is allowed for that service and that allowed downtime is also what's called an error budget:

In SRE, a team can "spend" that error budget on making unreliable changes. So basically that error budget says that we're allowed to have that much downtime in our system without losing business, making customers unhappy, etc.

Way to regulate the release speed ⚖️

So SLA is like a barometer, you can turn it up or down based on how reliable your system needs to be. And of course the closer to 100, the more effort you need to put in to guarantee the reliability of your systems.

Now once the SLA is defined the system performance can be measured against this number.

⛔️ If systems are more unreliable than the SLA allows: Then more resources from the SRE team will be put to make the systems reliable. Again because we have exceeded that allowed amount of downtime. And until the system is recovered to be within the defined SLA, fewer changes will be allowed:

✅ On the other hand if the system is performing well beyond the defined SLA: Developers in the SRE team can release more changes.

So it's a simple way to regulate the release speed of developers. ⚖️ If we turn up the SLA, releases will slow down and vice versa:

SRE Tasks and Responsibilities 👩‍💻

1) Automation - Create automated processes for operational aspects 👏

The SRE or the Site Reliability Engineer is the one, who creates automated processes to calculate and evaluate, whether the service is within the SLA or not.

So now the policy for launching is not the endless checklist that operations use to decide, whether to launch or not, instead an SRE helped design processes that can automatically evaluate an SLA. ✅

2) Configure Monitoring and Logging (Observability for System Performance) 🔎📈

Now of course to measure the performance of our systems and whether services are within the SLA, we need proper monitoring of our systems.

So as another big part of SRE tasks and responsibilities is to configure proper monitoring and logging of the systems to get visibility of what's going on inside.

3) Configure Monitoring and Logging (Observability for Detecting Issues) 🔎📉

Now we said that for most applications SLA is not 100%, which means we accept that it won't work 100% of the time. So at some point we will have an outage 😱

Now the question is, what do we do when an outage happens? Or how do we prepare for it? And that's where another big part of SRE tasks and responsibilities comes in.

The first one is monitoring and alerting, which I already mentioned. So in addition to giving you a visibility to measure your system's performance, but more importantly it helps you detect any indications for issues, before they happen or very early when they happen and then alert the teams about it:

4) Develop custom services to achieve this 👩‍💻

Now another important part of this whole configuration is that, when the issue is alerted and ideally the right person in the team gets the message, the alert message should contain all the needed information to identify and fix the issue fast.

Instead of:
❌ "something is wrong in the cluster"
a more detailed message like:
✅ "service a in cluster b is throwing 500 error"

So know you know exactly:
👉 which service is having a problem
👉 in which cluster and
👉 what that problem exactly is

So more detail the alert message, the better!

In many cases SREs will develop their own custom services to achieve this proper monitoring alerting and logging configuration for their systems.

5) Do On-Call Support ☎️

Another thing that SREs do is on-call support.

Basically, when things go wrong and users need real-time support, somebody is responsible for doing that and that is the on-call support team. And putting SREs on this support team has several benefits:

• It helps them really see and understand what issues to expect
• How the support deals with the issues?
• And what improvements can be made to make the support process more efficient?

Such as:

• do alert messages and logs have enough information to quickly identify the issue and the cause?
• Where issues identified too late?
• etc

So overall the main goal of SRE is to make sure the scope of outage is small when it happens, which means

1. 👍 the outage doesn't last long and is fixed very quickly
2. 👍 and less people and few services are affected by that outage

6) Post-Incident Reviews 🧐

Now fixing an issue or an outage is not the end of the SRE team's work. We want to use that outage as a chance for lessons learned and of course avoid that this happens in the future.

So as a principle of SRE is, to do, what's called Post Mortem, which is latin for "after death". So in SRE terms "after issue" or "after outage analysis".

This includes a thorough analysis, meaning taking time to really go deeper and understand the issues:

But of course during this analysis it's super important to stay blameless, which is one of the major points of this post-mortem analysis, in order to encourage people to admit and learn from their and other people's mistakes ✅

And finally it's important to document everything for future reference!

Who is doing SRE? SRE Role 👤

Now you may be thinking:
"How much more should software developers learn? They already have to know all these software development technologies, now they also have to take over the operations tasks and learn all these operations tools?" 🤯

SRE as its own Role

That's why we have SRE as its own role. So a dedicated person, whose full-time responsibility is to work on keeping systems reliable:

So in many projects along with developers, you have SRE teams, which are the new Operations team. Basically a team that does the operations, where both teams work for the same goal of keeping the systems within the defined SLAs:

So the SRE team maintains and takes care of the automated delivery operations and all sorts of automation that will help developers release their changes safely and fast. 🚀

Software Developers as SREs

However, it's also common to have 1 team of SREs and Software Developers, where SREs also do the software development job. And this means that Site Reliability Engineers must know software development as well, unlike DevOps engineers.

But in both cases, as you see we, started off with a traditional way of software development with separate dev and ops with opposing incentives and with SRE we gave devs and ops the same incentives and put them on the same side. 💙

SRE vs DevOps

Finally, one of the majorly discussed questions in this area is: What is the difference between SRE and DevOps Engineer or generally between these two concepts?

If you already watched my "What is DevOps" video, you know that there are two definitions of DevOps:

• the original definition, which is more high level and more broad and doesn't specify how exactly DevOps should be implemented
• and a more practical one, which evolved over time with its own DevOps engineer role.

So when we compare DevOps with SRE it's important to know which definition of DevOps we're using for this comparison?

1. First broader definition of DevOps vs. SRE
DevOps is a more high-level concept that defines what needs to be done to achieve the automated streamlined release process, while SRE is more specific about how to exactly implement this process and how to implement DevOps principles.

So many people would say that SRE is a specific implementation of the DevOps concepts:

2. Practical DevOps vs. SRE
But as we saw DevOps itself also became more practical with its own role and specific technologies and ways to implement it. So what's the comparison here? 🤔
Well, in many companies this practical DevOps implementation became more focused and concentrated on the speed of delivery for the application changes and of course even though it's part of the DevOps principles to not only release fast, but release quality code many DevOps teams in practice again seemed to optimize more for the speed than reliability.

So as a great complementary part of DevOps, SRE emerged with the same principles and goals in mind, which is release quality code fast, but as the name suggests more focused on reliability and keeping systems stable, while allowing for fast changes:

So SRE is its own role with its own set of tools for making systems reliable. So these two were kind of parallel developments and are now often seen as two sides of the same coin and it's not uncommon for teams to have both DevOps engineer and SRE helping implement the DevOps principles.

Now I hope you learned a lot in this article and that I was able to answer all of your questions about SRE. 🙂

---

Like, share and follow me 😍 for more content:

• YouTube
• Instagram - for behind-the-scenes and previews
• Twitter
• Private FB group
• LinkedIn