Avoid alert fatigue: alarms worth waking up for

Alert fatigue is the moment when an alarm triggers nothing more than a tired swipe across your phone. If you have ever been on call, you know the pattern: You take the first alarm at night seriously. By the tenth false alarm in the same week, however, you only check whether it turns green again on its own.

This topic matters to me because it is where engineering and leadership meet so directly. After all, a noisy alarm costs sleep and trust, and at some point the one notification that really mattered gets lost. That is why this article gives you both: a simple test to judge every alarm, and a step-by-step guide to building alarms in Amazon CloudWatch with the Serverless Framework so that they only wake you for real problems.

In short

Alert fatigue happens when many alarms require no action and the team learns to ignore them. A strict actionability test, separate channels for paging and tickets, and alarms on error budget consumption instead of single outliers help the most. The biggest pitfall is only ever adding alarms and never cleaning them up.

What you will learn in this article

• Spot alert fatigue in your team before on-call breaks down
• Use four questions to decide: page, ticket or delete
• Combine M out of N, missing data and composite alarms in CloudWatch
• Build a burn rate alarm on your error budget, including serverless.yml
• Run a monthly alarm review with a small TypeScript script

What is alert fatigue?

Alert fatigue describes becoming desensitized to warnings. You may know the term from hospitals, where constantly beeping devices cause exactly this problem. Software teams, too, experience something similar: Every alarm that nobody needs to act on teaches the team to ignore alarms. Therefore, this has little to do with a lack of discipline. Instead, it is a perfectly normal learning effect.

I find the four measures that Google describes for alerting strategies in the SRE Workbook helpful. Precision is the share of alerts that point to a real problem. Recall is the share of real problems that triggered an alert at all. On top of that there is detection time, meaning how quickly an alert fires, and reset time, meaning how long it keeps firing after the problem has ended.

Alert fatigue is therefore mostly a precision problem: too many notifications with nothing behind them. However, a long reset time hurts as well: An alarm that is still red an hour after the incident feels like a false alarm and gets treated like one next time.

How to spot alert fatigue in your team

Hardly anyone says in the daily: “I ignore our alarms.” Instead, the signs are quieter. For example, watch out for these signals:

• Alarms are acknowledged without anyone looking at logs or dashboards.
• In the alarm channel, nobody writes anymore what they found or did.
• The most frequent alarm of the past weeks “resolves itself”.
• People are reluctant to swap on-call weeks because everyone knows what the nights look like.
• After a real incident it turns out: The alarm fired, but nobody reacted.

Above all, the last one is expensive. If you are setting up an on-call rotation right now, also read my guide On-call rotation: how to organize it fairly. After all, a fair rotation does not help much if every shift is wrecked by false alarms.

The actionability test for every alarm

Before you tweak thresholds, you need a rule for which alarm may page anyone at all. I recommend four questions that you ask every existing and every new alarm:

1. Is anyone affected? First, do customers or other teams notice something right now, or will they soon?
2. Does someone need to act now? Will it get worse if nobody does anything until tomorrow morning?
3. Does the person know what to do? Is there a runbook, and is the link in the alarm text?
4. Has the alarm led to action recently? Was there at least one case in the past 30 days where someone actually stepped in?

The evaluation is deliberately strict. When nobody is affected, the metric belongs on a dashboard or can go entirely. However, if someone is affected but it is not urgent, it becomes a ticket for the next working day. And when the runbook is missing, it gets written before the alarm may page anyone at night. I show how to structure one in How to write a runbook. Finally, the fourth question is the reality check for the monthly review: An alarm that has only been triggered and dismissed for weeks loses its place in the pager.

The actionability test as a decision tree: Only alarms that pass every check may page someone at night.

Page, ticket or dashboard

As a result, the test leads to three channels. On AWS, this usually means two separate SNS topics for pages and tickets, and everything else only goes on a dashboard. SNS (Simple Notification Service) is the notification service that CloudWatch alarms send their messages to.

My rule of thumb: Each service only has a handful of paging alarms, and each of them describes a symptom that users feel. Causes like high CPU load or full queues are still valuable, but only as context. Because once the symptom is right, you will find the cause quickly enough on the dashboard.

Avoid alert fatigue with CloudWatch: step by step

The example is a fictional bike rental. A Lambda function createRental accepts new rentals behind an HTTP API. The goal, the SLO (service level objective), reads: 99.9 percent of invocations succeed. Therefore, the error budget is 0.1 percent.

Prerequisites

• an AWS account and the AWS CLI with valid credentials
• Node.js 22 and npm
• Serverless Framework v4 (npm install -g serverless); on first start, v4 asks you to log in
• basic Lambda knowledge; a good starting point is Monitoring Lambda: alarms, dashboard and budget with CDK

Step 1: Create the project and two channels

First, the opening part of serverless.yml defines the function and two SNS topics. The page topic is for everything that may wake someone up, while the ticket topic collects everything that can wait until tomorrow. In the example, both send email; in practice, however, you connect your on-call tool to page.

# serverless.yml
service: bike-rental-alerts
frameworkVersion: '4'

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-central-1
  memorySize: 512

params:
  default:
    pagerEmail: oncall@example.com
    ticketEmail: team@example.com
    # SLO: 99.9 % successful invocations -> error budget 0.1 %
    errorBudget: 0.001

functions:
  createRental:
    handler: src/handler.createRental
    events:
      - httpApi:
          method: post
          path: /rentals

resources:
  Resources:
    # Two channels: "page" wakes someone up, "ticket" waits until the morning
    PagerTopic:
      Type: AWS::SNS::Topic
      Properties:
        TopicName: ${self:service}-${sls:stage}-page
        Subscription:
          - Protocol: email
            Endpoint: ${param:pagerEmail}
    TicketTopic:
      Type: AWS::SNS::Topic
      Properties:
        TopicName: ${self:service}-${sls:stage}-ticket
        Subscription:
          - Protocol: email
            Endpoint: ${param:ticketEmail}

The function itself is deliberately simple so the focus stays on the alarms:

// src/handler.ts
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "./types.js";

export const createRental = async (
  event: APIGatewayProxyEventV2,
): Promise<APIGatewayProxyResultV2> => {
  const body = JSON.parse(event.body ?? "{}") as { bikeId?: string };
  if (!body.bikeId) {
    return { statusCode: 400, body: JSON.stringify({ message: "bikeId is required" }) };
  }
  // Demo only: a real function would store the rental in DynamoDB
  return { statusCode: 201, body: JSON.stringify({ rentalId: crypto.randomUUID(), bikeId: body.bikeId }) };
};

// src/types.ts
// Minimal types so the example needs no extra package
export interface APIGatewayProxyEventV2 {
  body?: string;
}
export interface APIGatewayProxyResultV2 {
  statusCode: number;
  body: string;
}

Step 2: Burn rate instead of a fixed error threshold

The classic alarm “Errors greater than 0” is the fastest route to alert fatigue. After all, a single error at three in the morning is no reason to wake anyone up. Instead, a better question is: How fast are we consuming our error budget right now?

That is exactly what the burn rate measures. With a burn rate of 1, the budget is used up exactly at the end of the SLO period, so for 99.9 percent over 30 days that means a constant error rate of 0.1 percent. In other words, a burn rate of 14.4 means: Within one hour, 2 percent of the monthly budget is gone, and if it continues, the entire budget is used up after a little more than two days. For an SLO of 99.9 percent over 30 days, the SRE Workbook recommends this combination:

In the example, I therefore implement the first row. Two alarms calculate the burn rate with metric math (expressions over CloudWatch metrics): errors divided by invocations divided by the budget. One alarm looks at one hour, the other at five minutes.

# serverless.yml (continued under resources.Resources)
    # Burn rate over 1 hour (long window): no actions of its own
    BurnRateLongAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-burn-1h
        AlarmDescription: Error budget burn rate over 1 hour above 14.4
        ComparisonOperator: GreaterThanThreshold
        Threshold: 14.4
        EvaluationPeriods: 1
        DatapointsToAlarm: 1
        TreatMissingData: notBreaching
        Metrics:
          - Id: burn
            Label: Burn rate 1h
            Expression: IF(inv >= 20, FILL(err, 0) / inv / ${param:errorBudget}, 0)
            ReturnData: true
          - Id: err
            ReturnData: false
            MetricStat:
              Stat: Sum
              Period: 3600
              Metric:
                Namespace: AWS/Lambda
                MetricName: Errors
                Dimensions:
                  - Name: FunctionName
                    Value:
                      Ref: CreateRentalLambdaFunction
          - Id: inv
            ReturnData: false
            MetricStat:
              Stat: Sum
              Period: 3600
              Metric:
                Namespace: AWS/Lambda
                MetricName: Invocations
                Dimensions:
                  - Name: FunctionName
                    Value:
                      Ref: CreateRentalLambdaFunction

    # Burn rate over 5 minutes (short window): ends the alert soon after recovery
    BurnRateShortAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-burn-5m
        AlarmDescription: Error budget burn rate over 5 minutes above 14.4
        ComparisonOperator: GreaterThanThreshold
        Threshold: 14.4
        EvaluationPeriods: 1
        DatapointsToAlarm: 1
        TreatMissingData: notBreaching
        Metrics:
          - Id: burn
            Label: Burn rate 5m
            Expression: IF(inv >= 5, FILL(err, 0) / inv / ${param:errorBudget}, 0)
            ReturnData: true
          - Id: err
            ReturnData: false
            MetricStat:
              Stat: Sum
              Period: 300
              Metric:
                Namespace: AWS/Lambda
                MetricName: Errors
                Dimensions:
                  - Name: FunctionName
                    Value:
                      Ref: CreateRentalLambdaFunction
          - Id: inv
            ReturnData: false
            MetricStat:
              Stat: Sum
              Period: 300
              Metric:
                Namespace: AWS/Lambda
                MetricName: Invocations
                Dimensions:
                  - Name: FunctionName
                    Value:
                      Ref: CreateRentalLambdaFunction

Two details matter. First, FILL(err, 0) sets missing error values to 0. In addition, IF(inv >= 20, …, 0) protects you from a classic: If only three requests arrive at night and one of them fails, the error rate is 33 percent and the burn rate is far above 300. Because of the minimum number of invocations, the alarm stays quiet while there is too little traffic for a reliable statement. Finally, adjust the values 20 and 5 to your traffic.

Both alarms deliberately have no AlarmActions. Instead, they are only building blocks for step 4.

Step 3: Set M out of N and missing data deliberately

Not every metric fits an SLO. For latency, for example, a ticket alarm that ignores single outliers is often enough. For this, CloudWatch offers M out of N evaluation for this: EvaluationPeriods is N, meaning how many of the most recent data points are considered. DatapointsToAlarm is M, meaning how many of them must be above the threshold. However, the breaching points do not have to be consecutive.

# serverless.yml (continued)
    # Latency: only 4 of 5 breaching minutes count, single outliers do not
    LatencyAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-latency-p95
        AlarmDescription: p95 duration above 2 seconds in 4 of the last 5 minutes
        Namespace: AWS/Lambda
        MetricName: Duration
        Dimensions:
          - Name: FunctionName
            Value:
              Ref: CreateRentalLambdaFunction
        ExtendedStatistic: p95
        Period: 60
        EvaluationPeriods: 5
        DatapointsToAlarm: 4
        Threshold: 2000
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching
        AlarmActions:
          - Ref: TicketTopic

Thus, with four out of five minutes above two seconds, the alarm reacts to a real latency problem, but not to a single cold start. In addition, the second lever is TreatMissingData. It only applies when CloudWatch has too few real data points for the evaluation:

Lambda emits no duration values without invocations. For the bike rental, a quiet night therefore means no data, and that is fine. That is why this alarm uses notBreaching. One exception to the defaults: According to the AWS documentation, alarms in the AWS/DynamoDB namespace default to ignore.

Step 4: Composite alarm with a pause during deployments

Next, the building blocks come together. A composite alarm does not evaluate metrics; it only looks at the states of other alarms. The rule ALARM(long) AND ALARM(short) ensures that someone only gets paged when the budget burns too fast over both the last hour and the last five minutes.

Above all, the short window improves the reset time. Once the incident is over, the five-minute burn rate quickly drops below the threshold, the composite alarm returns to OK, and the on-call person gets an all-clear. Otherwise, with only the one-hour window, the alarm would stay red for a long time.

🎬 Burn rate alarm with two time windows – this animated diagram only runs on my site: watch it on sascha-ms.de

A second tool against alert fatigue is action suppression. During a deployment, for example, short error spikes are normal and nobody needs to be woken up by them. The DeploymentInProgress alarm watches a custom metric that your pipeline writes.

# serverless.yml (continued)
    # Deployment marker: in ALARM while the pipeline reports a deployment
    DeploymentInProgressAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-deployment-in-progress
        AlarmDescription: Set by the deployment pipeline, suppresses pages
        Namespace: BikeRental/Ops
        MetricName: DeploymentInProgress
        Statistic: Maximum
        Period: 900
        EvaluationPeriods: 1
        Threshold: 1
        ComparisonOperator: GreaterThanOrEqualToThreshold
        TreatMissingData: notBreaching

    # The only alarm allowed to wake someone up
    PageCompositeAlarm:
      Type: AWS::CloudWatch::CompositeAlarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-page-error-budget
        AlarmDescription: >-
          Error budget is burning 14.4 times faster than allowed (1h and 5m).
          Runbook: https://example.com/runbooks/bike-rental-errors
        AlarmRule:
          Fn::Join:
            - ''
            - - 'ALARM("'
              - Ref: BurnRateLongAlarm
              - '") AND ALARM("'
              - Ref: BurnRateShortAlarm
              - '")'
        ActionsSuppressor:
          Ref: DeploymentInProgressAlarm
        ActionsSuppressorWaitPeriod: 120
        ActionsSuppressorExtensionPeriod: 600
        AlarmActions:
          - Ref: PagerTopic
        OKActions:
          - Ref: PagerTopic

The three suppressor fields work like this: While the DeploymentInProgress alarm is in the ALARM state, the composite alarm performs no actions. ActionsSuppressorWaitPeriod defines how many seconds the composite alarm waits for the suppressor to switch to ALARM. ActionsSuppressorExtensionPeriod extends the pause after the suppressor has left ALARM. However, if the problem still exists afterwards, the composite alarm performs its actions. As a result, a broken deployment does not go unnoticed.

The period of 900 seconds keeps the marker alarm in the ALARM state for a while after the last data point. Therefore, choose it a little longer than your typical deployment. Then, in the pipeline, you write a data point before the deployment:

aws cloudwatch put-metric-data \
  --namespace BikeRental/Ops \
  --metric-name DeploymentInProgress \
  --value 1
serverless deploy --stage prod

Step 5: Deploy and test the alarm

Install the dependencies and deploy with your own email addresses. Serverless Framework v4 builds TypeScript handlers automatically with esbuild.

npm install
serverless deploy --stage dev \
  --param="pagerEmail=you@example.com" \
  --param="ticketEmail=team@example.com"

Afterwards you receive two emails from AWS Notifications asking you to confirm the subscription. Without confirmation, nothing arrives later. Next, to test the chain, set both child alarms to ALARM manually:

aws cloudwatch set-alarm-state \
  --alarm-name bike-rental-alerts-dev-burn-1h \
  --state-value ALARM --state-reason "Testing the alarm chain"
aws cloudwatch set-alarm-state \
  --alarm-name bike-rental-alerts-dev-burn-5m \
  --state-value ALARM --state-reason "Testing the alarm chain"
aws cloudwatch describe-alarms --alarm-types CompositeAlarm \
  --query "CompositeAlarms[].[AlarmName,StateValue]" --output table

Expected result: The composite alarm bike-rental-alerts-dev-page-error-budget is in ALARM, and a notification goes to the page address. Afterwards, at the next evaluation, the child alarms return to OK based on the real metrics, and you receive the OK message. If no email arrives, first check the subscription confirmation and then whether the deployment marker is still in ALARM.

Monthly alarm review against alert fatigue

Alarms age, because traffic changes, functions get added, and last year's threshold no longer fits. That is why every alerting strategy needs a fixed ritual: 30 minutes once a month with everyone on the on-call rotation. CloudWatch keeps alarm history for 30 days, so that fits perfectly.

So that nobody has to count state changes by hand, a small TypeScript script reads the history via DescribeAlarmHistory. For each alarm, it counts how often it fired, how often at night, and how often it resolved itself within ten minutes.

// src/review.ts
export type AlarmState = "OK" | "ALARM" | "INSUFFICIENT_DATA";

export interface StateChange {
  alarmName: string;
  timestamp: Date;
  oldState: AlarmState;
  newState: AlarmState;
}

export interface AlarmReport {
  alarmName: string;
  alarms: number;
  nightAlarms: number;
  selfResolved: number;
  medianMinutesInAlarm: number | null;
  verdict: "noise" | "check" | "ok";
}

// Parses the HistoryData JSON of a StateUpdate entry from DescribeAlarmHistory
export function parseHistoryData(alarmName: string, timestamp: Date, historyData: string): StateChange | null {
  try {
    const data = JSON.parse(historyData) as {
      oldState?: { stateValue?: AlarmState };
      newState?: { stateValue?: AlarmState };
    };
    const oldState = data.oldState?.stateValue;
    const newState = data.newState?.stateValue;
    if (!oldState || !newState) return null;
    return { alarmName, timestamp, oldState, newState };
  } catch {
    return null;
  }
}

function localHour(date: Date, timeZone: string): number {
  const hour = new Intl.DateTimeFormat("en-GB", { hour: "numeric", hourCycle: "h23", timeZone }).format(date);
  return Number(hour);
}

function median(values: number[]): number | null {
  if (values.length === 0) return null;
  const sorted = [...values].sort((a, b) => a - b);
  const mid = Math.floor(sorted.length / 2);
  return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
}

export interface ReviewOptions {
  timeZone?: string;
  nightStartHour?: number;
  nightEndHour?: number;
  selfResolvedMinutes?: number;
}

export function buildReport(changes: StateChange[], options: ReviewOptions = {}): AlarmReport[] {
  const { timeZone = "Europe/Berlin", nightStartHour = 22, nightEndHour = 7, selfResolvedMinutes = 10 } = options;

  const byAlarm = new Map<string, StateChange[]>();
  for (const change of changes) {
    const list = byAlarm.get(change.alarmName) ?? [];
    list.push(change);
    byAlarm.set(change.alarmName, list);
  }

  const reports: AlarmReport[] = [];
  for (const [alarmName, list] of byAlarm) {
    list.sort((a, b) => a.timestamp.getTime() - b.timestamp.getTime());
    let alarms = 0;
    let nightAlarms = 0;
    let selfResolved = 0;
    const durations: number[] = [];

    list.forEach((change, index) => {
      if (change.newState !== "ALARM") return;
      alarms++;
      const hour = localHour(change.timestamp, timeZone);
      if (hour >= nightStartHour || hour < nightEndHour) nightAlarms++;

      const end = list.slice(index + 1).find((next) => next.oldState === "ALARM");
      if (end) {
        const minutes = (end.timestamp.getTime() - change.timestamp.getTime()) / 60_000;
        durations.push(minutes);
        if (minutes <= selfResolvedMinutes) selfResolved++;
      }
    });

    const verdict: AlarmReport["verdict"] =
      alarms >= 3 && selfResolved / alarms >= 0.5 ? "noise" : nightAlarms > 0 ? "check" : "ok";
    reports.push({ alarmName, alarms, nightAlarms, selfResolved, medianMinutesInAlarm: median(durations), verdict });
  }

  return reports.sort((a, b) => b.alarms - a.alarms);
}

// scripts/alarm-review.ts
import { CloudWatchClient, DescribeAlarmHistoryCommand } from "@aws-sdk/client-cloudwatch";
import { buildReport, parseHistoryData, type StateChange } from "../src/review.js";

const days = Number(process.argv[2] ?? "30"); // CloudWatch keeps alarm history for 30 days
const client = new CloudWatchClient({});

async function loadStateChanges(): Promise<StateChange[]> {
  const changes: StateChange[] = [];
  let nextToken: string | undefined;
  do {
    const page = await client.send(
      new DescribeAlarmHistoryCommand({
        HistoryItemType: "StateUpdate",
        AlarmTypes: ["MetricAlarm", "CompositeAlarm"],
        StartDate: new Date(Date.now() - days * 24 * 60 * 60 * 1000),
        EndDate: new Date(),
        MaxRecords: 100,
        NextToken: nextToken,
      }),
    );
    for (const item of page.AlarmHistoryItems ?? []) {
      if (!item.AlarmName || !item.Timestamp || !item.HistoryData) continue;
      const change = parseHistoryData(item.AlarmName, item.Timestamp, item.HistoryData);
      if (change) changes.push(change);
    }
    nextToken = page.NextToken;
  } while (nextToken);
  return changes;
}

const report = buildReport(await loadStateChanges());
console.table(report);

Run the review script

The classification is deliberately simple. For example, if an alarm fired at least three times and at least half of those were over after ten minutes, it lands on the list as noise. In contrast, an alarm with night-time triggers gets check. In addition, you need a package.json with "type": "module" and this tsconfig.json:

// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "outDir": "dist",
    "rootDir": ".",
    "skipLibCheck": true
  },
  "include": ["src", "scripts", "test"]
}

npm install @aws-sdk/client-cloudwatch
npm install --save-dev typescript @types/node
npx tsc && node dist/scripts/alarm-review.js 30

As a result, you get a table with the columns alarmName, alarms, nightAlarms, selfResolved, medianMinutesInAlarm and verdict, sorted by frequency. Finally, the script needs a single IAM permission: cloudwatch:DescribeAlarmHistory.

Agenda for the review meeting

Then, in the review meeting, you proceed like this:

1. First, go through the five most frequent alarms and apply the actionability test to each.
2. Make a decision for every noise alarm: M out of N, a higher threshold, the ticket channel or deletion.
3. For every night-time alarm, ask: Would tomorrow morning have been soon enough?
4. Look for gaps: Which incident did a customer report before any alarm fired?
5. Finally, record every change with an owner and a date so it does not gather dust in the notes.

The fourth point matters, because fighting alert fatigue does not mean blindly deleting alarms. Recall counts just as much as precision.

What the alarms cost and what you should secure

According to the CloudWatch pricing page, a standard alarm in US East costs 0.10 US dollars per alarm metric per month. For metric math alarms, however, every metric in the expression counts, so a burn rate alarm with errors and invocations costs 0.20 US dollars. A composite alarm costs 0.50 US dollars per month. The example therefore comes to roughly 1.10 US dollars per function per month; prices vary slightly by region. In addition, the free tier covers ten alarm metrics for standard alarms that reference metrics directly.

There are three points regarding security. First, alarm texts end up in emails and chat tools, so they should contain runbook links but no credentials or customer data. Second, for the deployment marker, the pipeline only needs cloudwatch:PutMetricData, and you can restrict it to BikeRental/Ops with the cloudwatch:namespace condition. Third, the review script only needs read access.

Common mistakes and how to avoid them

• Errors greater than 0 as a paging alarm: pages on every single error. Use burn rate or at least M out of N.
• Actions on child alarms and the composite alarm: leads to duplicate notifications. Only the composite alarm gets actions.
• breaching for metrics that only report values when something goes wrong: makes alarms fire on quiet nights.
• Burn rate without minimum traffic: at night, a few errors are enough for absurd values. The IF condition prevents this.
• Composite alarms that reference each other: According to AWS, such cycles are no longer evaluated and cannot be deleted. Setting one alarm rule to FALSE breaks the cycle.
• No runbook in the alarm text: The person woken up at night has to search first. The link belongs in AlarmDescription.

My take

For me, alert fatigue is a leadership topic first and a configuration topic second. Therefore, whoever is on call must have the right to change or delete alarms without asking for permission. I also think one simple rule makes sense: No new paging alarm without a runbook and without a person who owns it.

I disagree with the saying “Better one alarm too many than one too few.” In theory, it sounds careful. In practice, however, every unnecessary alarm lowers the chance that the important one is taken seriously. I am equally skeptical when an AI agent is supposed to pre-sort noisy alarms. Although it can help with triage, the noise remains, and the team gets used to following the pre-sorting blindly. More on that in Automation bias: when teams trust AI blindly.

Burn rate alarms are not the right answer for every service. With very little traffic, for example, the numbers are too thin. Instead, synthetic checks that regularly send a real request, or a ticket alarm on absolute error counts, work better there.

FAQ

What is alert fatigue?

Alert fatigue is becoming desensitized to alarms when too many notifications require no action. As a result, the team reacts more slowly or not at all, even to real problems.

How many alarms are too many?

There is no fixed number. Instead, a good indicator is the share of pages that lead to real action. If it is below half, in my view you have a noise problem.

How much does a composite alarm cost in CloudWatch?

In US East, a composite alarm costs 0.50 US dollars per month, regardless of how many alarms it combines. In addition, the child alarms are charged separately per alarm metric.

Can I mute CloudWatch alarms during a deployment?

Yes. Composite alarms offer ActionsSuppressor with a wait and an extension period. Alternatively, a rule such as NOT ALARM(DeploymentInProgress) excludes that period, or you temporarily disable actions with aws cloudwatch disable-alarm-actions.

Your next step

This week, take the five most frequent alarms of the past 30 days and ask each of them the four questions from the actionability test. Usually, one or two candidates then stand out right away that should never have paged anyone. And now I am curious: Which alarm last woke you up for nothing, and what did you do with it afterwards?

Sources

• Google SRE Workbook: Alerting on SLOs
• Amazon CloudWatch: Alarm evaluation
• Amazon CloudWatch: Configuring how alarms treat missing data
• Amazon CloudWatch: Create a composite alarm
• CloudWatch API Reference: PutCompositeAlarm
• CloudWatch API Reference: DescribeAlarmHistory
• Amazon CloudWatch: Using alarms
• Amazon CloudWatch: Using metric math
• Amazon CloudWatch Pricing

This post was created with AI assistance.

👉 Next step: Working with AWS CDK? Grab my printable AWS CDK cheat sheet with commands, concepts and a stack skeleton.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en