How to Write a Runbook: Template and Lambda Examples

Sitting down to write a runbook feels like busywork you can always push to “later”. Until an alarm goes off at three in the morning, the colleague who knows the system is on vacation, and you are half asleep trying to remember how to send messages back from a dead-letter queue.

The topic matters to me because it sits right where technology and team culture meet. Knowledge that lives in only two heads is an operational risk and a burden for exactly those two people. In my article on alert fatigue I set a rule: an alarm may only wake someone up at night once there is a runbook for it. This is the second half: what such a runbook looks like, how to write it step by step, and how a small check in your pipeline stops alarms from going live without instructions.

In short

If you want to write a runbook that works, start with a clear goal. A runbook is a step-by-step guide with a defined outcome that someone without deep knowledge of the service can follow at night. Copyable commands, actions written as if-then decisions and a verification step that proves the problem is really gone help the most. The biggest pitfall is a runbook that was written once and never tested by anyone else.

What you will learn in this article

• How a runbook differs from a playbook and from an automation runbook
• Which sections keep a runbook usable even at three in the morning
• How to write a runbook for a full DLQ in seven steps
• How serverless.yml puts the runbook link straight into the CloudWatch alarm
• How a TypeScript check in your pipeline blocks alarms without a maintained runbook

What is a runbook?

A runbook is a documented sequence of steps that leads to a defined outcome. In its simplest form, the AWS Well-Architected Framework describes it as a checklist for completing a task. The outcome is what counts: “The DLQ is empty and every order has been processed” is a goal. “Notes on the ticket queue” is not.

You will run into two related terms. A playbook helps you investigate: which logs do I check, which metrics, which hypotheses do I test first? AWS draws the line by saying that only the runbook has a desired outcome. Often it comes into play after the playbook has found the root cause. In practice the boundary blurs, and I am fine with that. I put the most important diagnosis steps straight into the runbook so nobody has to jump between two documents at night.

The third term comes from AWS itself. In AWS Systems Manager, automated procedures are called runbooks as well: YAML or JSON documents whose steps the service runs through API calls. That is the advanced stage. This article starts one level earlier, with a plain-text runbook in your repository. Without a clearly described procedure, there is nothing meaningful to automate anyway.

Why you should write a runbook before the alarm wakes you

The Google SRE book contains a figure I like to bring into discussions: procedures written down in advance produce roughly a “3x improvement in MTTR” compared with improvising. MTTR stands for mean time to repair, the average time it takes to fix an issue. The number comes from Google’s operations and does not translate one to one to a team of eight. The direction still makes sense. If you do not have to think about which command is the right one at night, you are faster and make fewer mistakes.

The human reasons matter at least as much to me. A runbook spreads knowledge that would otherwise sit with the two most experienced people on the team. Without a runbook, those two get called for every incident, even on vacation. New team members can join the on-call rotation sooner because they do not need everything in their heads. On top of that, writing it uncovers gaps: by the time you reach “How do I know it worked?”, you often notice there is no metric for it at all.

Of course a runbook does not replace thinking. The SRE book stresses that too: no playbook, however good, replaces people who react smartly in the moment. The runbook takes care of the routine so your mind stays free for whatever is genuinely new.

🎬 From alarm to green with a runbook – this animated diagram only runs on my site: watch it on sascha-ms.de

What belongs in a good runbook

The template in the Well-Architected Framework is deliberately lean: ID, description, tools, special permissions, author, last updated, escalation contact, and then the steps. For Lambda services I would add a few sections that make the difference at night.

Anatomy of a runbook: eleven sections that make the difference at night.

Two sections get forgotten most often: verify and rollback. Without a verification step, an incident ends as soon as someone types “looks good” in the chat. Without a rollback, nobody dares to run the action because it is unclear what happens if it goes wrong.

Write a runbook in seven steps

The example is a made-up concert ticket shop called “Konzertkasse” (German for box office). A Lambda function named issueTickets reads paid orders from an SQS queue and sends out the tickets. If a message fails five times, it ends up in a dead-letter queue (DLQ), a holding area for messages that could not be processed. That is exactly the case we now want to write a runbook for.

Prerequisites

• A Git repository with a docs/runbooks folder
• AWS CLI v2 and a profile with the on-call role
• Serverless Framework v3 with the serverless-esbuild plugin (v4 has esbuild built in, so you can drop the plugin there)
• Node.js 22 for the runbook check
• Someone who did not build the service, for the test in step 7

Step 1: Pick the most important alarm

When you write your first runbook, do not start with all alarms at once. For getting started, the Well-Architected Framework recommends short, regular procedures with low risk so the team gets used to the format. I would still begin with the alarm that drags someone out of bed most often, because that is where the runbook pays off fastest. For the Konzertkasse, that is the full DLQ, since paying customers without tickets are behind it.

Step 2: Goal and impact in two sentences

First write down when the work is done, then who is affected. The goal must be verifiable, for example: “The DLQ is empty and the alarm is OK.” The impact helps with prioritizing. Anyone who reads “Customers have paid but received no ticket” while half asleep knows immediately that this cannot wait until morning.

Step 3: Diagnosis with copyable commands

Every diagnosis step gets a command you can copy without editing it. That is why you set the queue URL and ARN once in the preparation section. For the Konzertkasse it looks like this:

export AWS_PROFILE=konzertkasse-prod-oncall
export DLQ_URL=$(aws sqs get-queue-url --queue-name konzertkasse-prod-ticket-orders-dlq --query QueueUrl --output text)
export DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)

# How many messages are in the DLQ?
aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages

# Look at one message without deleting it
aws sqs receive-message --queue-url "$DLQ_URL" --max-number-of-messages 1 --visibility-timeout 0

The first command returns JSON with the field ApproximateNumberOfMessages. The fastest way to find the failure reason is a CloudWatch Logs Insights query on the function’s log group:

fields @timestamp, reason
| filter msg = "ticket_failed"
| stats count() by reason

This query only works because the function writes its errors as JSON with fixed fields. So a runbook nudges you toward better logging along the way. The handler is deliberately small and reports failed messages individually so only those get retried:

// src/issue-tickets.ts
import type { SQSBatchResponse, SQSEvent } from 'aws-lambda';

interface TicketOrder {
  orderId: string;
  concertId: string;
  seats: number;
}

function parseOrder(body: string): TicketOrder {
  const data = JSON.parse(body) as Partial<TicketOrder>;
  if (!data.orderId || !data.concertId || !Number.isInteger(data.seats)) {
    throw new Error('Order incomplete');
  }
  return data as TicketOrder;
}

export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
  const batchItemFailures: SQSBatchResponse['batchItemFailures'] = [];

  for (const record of event.Records) {
    try {
      const order = parseOrder(record.body);
      // This is where the ticket would be created and emailed.
      console.log(JSON.stringify({ level: 'INFO', msg: 'ticket_issued', orderId: order.orderId }));
    } catch (error) {
      // The runbook query looks for exactly this msg value.
      console.log(JSON.stringify({
        level: 'ERROR',
        msg: 'ticket_failed',
        messageId: record.messageId,
        reason: (error as Error).message,
      }));
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures };
}

Step 4: Phrase actions as decisions

The most common mistake in runbooks is a single action for every situation. With a full DLQ, however, there are at least three situations, and each needs something different. That is why I phrase actions as “if …, then …”:

• If the cause is fixed (third-party system reachable again, fix deployed), you send the messages back to the main queue at a throttled rate.
• If the cause is still active or unclear, you send nothing back. The messages stay in the DLQ for up to 14 days, and you escalate.
• If a single message is broken (for example, a required field is missing), you note the message ID in the ticket and check with the product team whether it may be deleted.

aws sqs start-message-move-task --source-arn "$DLQ_ARN" --max-number-of-messages-per-second 10

Without a destination, SQS sends the messages back to their original queue. Throttling to 10 messages per second is intentional: if the third-party system has only just come back, the backlog should not knock it over again right away. If you leave the value out, SQS picks the rate itself. The maximum is 500 messages per second.

Step 5: Define verification and rollback

This part gets skipped a lot when people write a runbook. The verification step describes what you need to see before you declare the incident over. The rollback describes how to stop an action that is running. Both belong in the runbook as commands:

# Redrive progress: status RUNNING or COMPLETED
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN"

# Is the alarm back to OK?
aws cloudwatch describe-alarms --alarm-names konzertkasse-prod-ticket-orders-dlq-not-empty --query 'MetricAlarms[0].StateValue'

# Rollback: cancel a running redrive (TaskHandle from the start-message-move-task output)
aws sqs cancel-message-move-task --task-handle "<TaskHandle>"

Allow for some waiting time when verifying. The alarm evaluates five-minute periods, so it does not flip to OK instantly. Messages that were already moved stay in the main queue after a cancel, by the way, and that note belongs in the rollback section too.

Step 6: Put the runbook link into the alarm

A runbook you first have to search for will not get read at night. That is why the link goes straight into the AlarmDescription. This text ends up in the email or chat notification, and CloudWatch allows up to 1,024 characters there. The complete serverless.yml for the Konzertkasse looks like this:

# serverless.yml
service: konzertkasse
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-central-1
  stage: ${opt:stage, 'dev'}
  timeout: 10

custom:
  runbookBaseUrl: https://github.com/beispiel-org/konzertkasse/blob/main/docs/runbooks
  esbuild:
    format: esm
    outputFileExtension: .mjs

functions:
  issueTickets:
    handler: src/issue-tickets.handler
    events:
      - sqs:
          arn: !GetAtt TicketOrdersQueue.Arn
          batchSize: 10
          functionResponseType: ReportBatchItemFailures

resources:
  Resources:
    PagerTopic:
      Type: AWS::SNS::Topic

    TicketOrdersDlq:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: ${self:service}-${sls:stage}-ticket-orders-dlq
        MessageRetentionPeriod: 1209600 # 14 days to diagnose

    TicketOrdersQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: ${self:service}-${sls:stage}-ticket-orders
        VisibilityTimeout: 60 # six times the function timeout
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt TicketOrdersDlq.Arn
          maxReceiveCount: 5

    TicketOrdersDlqAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-ticket-orders-dlq-not-empty
        AlarmDescription: >-
          Orders are stuck in the DLQ, customers get no ticket.
          Runbook: ${self:custom.runbookBaseUrl}/rb-01-dlq-ticket-orders.md
        Namespace: AWS/SQS
        MetricName: ApproximateNumberOfMessagesVisible
        Dimensions:
          - Name: QueueName
            Value: !GetAtt TicketOrdersDlq.QueueName
        Statistic: Maximum
        Period: 300
        EvaluationPeriods: 1
        Threshold: 0
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching
        AlarmActions:
          - !Ref PagerTopic

    IssueTicketsErrorsAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${sls:stage}-issue-tickets-errors
        AlarmDescription: >-
          The issueTickets function keeps failing, often after a deploy.
          Runbook: ${self:custom.runbookBaseUrl}/rb-02-rollback-issue-tickets.md
        Namespace: AWS/Lambda
        MetricName: Errors
        Dimensions:
          - Name: FunctionName
            Value: ${self:service}-${sls:stage}-issueTickets
        Statistic: Sum
        Period: 60
        EvaluationPeriods: 5
        DatapointsToAlarm: 3
        Threshold: 5
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching
        AlarmActions:
          - !Ref PagerTopic

plugins:
  - serverless-esbuild

Two details matter here. At 60 seconds, the queue’s visibility timeout is six times the function timeout, as AWS recommends for SQS triggers. In addition, TreatMissingData: notBreaching makes sure missing data points do not count as a problem. Deploying takes one command:

npx serverless deploy --stage prod

At the end, the framework reports Service deployed to stack konzertkasse-prod. Costs stay small: in US East, two standard alarms cost USD 0.10 each per month, and the queue and function barely register at low load.

Step 7: Have someone else test the runbook

Hand the runbook to someone who did not build the service and let them walk through it in staging. You only watch and take notes wherever they get stuck. It sounds trivial, yet when you write a runbook this is the most valuable step, because everything that is obvious to you is missing from the text. To fill the DLQ, send a deliberately incomplete order:

export QUEUE_URL=$(aws sqs get-queue-url --queue-name konzertkasse-staging-ticket-orders --query QueueUrl --output text)
aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"orderId":"test-1"}'

After five failed attempts, which takes about five to six minutes, the message lands in the DLQ and the alarm fires. An exercise like this fits nicely into a game day. Afterwards you enter the date under “Last tested”.

The complete example runbook

This is what RB-01 looks like as a Markdown file in the repository. Feel free to use it as a starting point and swap in your own names, commands and thresholds:

# RB-01: Orders in the DLQ (ticket-orders)

| Runbook ID | Owner | Permissions | Escalation |
|---|---|---|---|
| RB-01 | Team Box Office | Role `oncall-konzertkasse` | On-call level 2 |

Last tested: 2026-11-02

## Goal
Every order from the DLQ has been processed or deliberately set aside. The DLQ is empty and the alarm is back to OK.

## Trigger
Alarm `konzertkasse-prod-ticket-orders-dlq-not-empty`: at least one message in the DLQ.

## Impact
Customers have paid but receive no ticket by email. New orders keep flowing as long as only a few messages fail.

## Preparation

bash
export AWS_PROFILE=konzertkasse-prod-oncall
export DLQ_URL=$(aws sqs get-queue-url --queue-name konzertkasse-prod-ticket-orders-dlq --query QueueUrl --output text)
export DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn --query Attributes.QueueArn --output text)

## Diagnosis
1. Check the volume:
   `aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages`
2. Find the failure reason in CloudWatch Logs Insights (log group `/aws/lambda/konzertkasse-prod-issueTickets`, last 60 minutes):
   `fields @timestamp, reason | filter msg = "ticket_failed" | stats count() by reason`
3. Look at one message without deleting it:
   `aws sqs receive-message --queue-url "$DLQ_URL" --max-number-of-messages 1 --visibility-timeout 0`

## Actions
- **Cause fixed** (third-party system reachable again, fix deployed): send the messages back, throttled to 10 per second.
  `aws sqs start-message-move-task --source-arn "$DLQ_ARN" --max-number-of-messages-per-second 10`
- **Cause still active or unclear**: do not send anything back. The messages stay for up to 14 days. If the error rate keeps rising, check RB-03 (pause processing) and escalate.
- **Single broken message** (e.g. a required field is missing): note the message ID in the ticket, clarify with the product team, delete only afterwards.

## Verify
- `aws sqs list-message-move-tasks --source-arn "$DLQ_ARN"` shows `Status: COMPLETED`.
- `ApproximateNumberOfMessages` of the DLQ is 0.
- `aws cloudwatch describe-alarms --alarm-names konzertkasse-prod-ticket-orders-dlq-not-empty --query 'MetricAlarms[0].StateValue'` returns `"OK"` (can take up to 5 minutes).

## Rollback
Stop a running redrive: `aws sqs cancel-message-move-task --task-handle "<TaskHandle>"`. Messages already moved stay in the main queue.

## Escalation
No progress after 30 minutes, more than 500 messages or suspected duplicate tickets: call level 2 and open the incident channel.

markdown

On security: a runbook in a repository never contains credentials. Message contents do not belong in a ticket or chat either if they include names or email addresses. For the redrive, the on-call role needs sqs:StartMessageMoveTask, sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the DLQ, plus sqs:SendMessage on the destination queue. If the queues are encrypted with KMS, add kms:Decrypt and kms:GenerateDataKey.

Three runbooks every Lambda service needs

Besides a full DLQ, two situations come up in almost every Serverless project sooner or later: a broken deploy and a function that is actively causing damage. Both deserve a runbook before things get serious.

Redriving a DLQ and where that does not work

Redriving with start-message-move-task only works for DLQs whose source is an SQS queue. If you configured a DLQ directly on Lambda for an asynchronously invoked function, SQS rejects it as a source. In that case the runbook needs a different path, such as a small script that reads the messages and invokes the function again. Also, only one redrive can run per queue at a time.

Rolling back after a broken deploy

When errors spike right after a deploy, going back is usually the fastest route. With the Serverless Framework, you list the recent deployments and roll back to a timestamp:

npx serverless deploy list --stage prod
npx serverless rollback --stage prod --timestamp <timestamp>

If you work with AWS CDK or with an alias such as live, point the alias at the previous version instead. Look up the version number first with aws lambda list-versions-by-function:

aws lambda update-alias --function-name konzertkasse-prod-issueTickets --name live --function-version 41

Important for the runbook: rolling back code does not undo data changes. If the deploy included a change to the data format, the runbook has to say so explicitly and, when in doubt, forbid the rollback.

Emergency brake for a function causing damage

Sometimes a function has to stop immediately, for instance because it sends duplicate tickets. For an SQS trigger, the cleanest brake is to disable the event source mapping, the connection between queue and function:

export ESM_UUID=$(aws lambda list-event-source-mappings --function-name konzertkasse-prod-issueTickets --query 'EventSourceMappings[0].UUID' --output text)
aws lambda update-event-source-mapping --uuid "$ESM_UUID" --no-enabled

# Turn it back on later
aws lambda update-event-source-mapping --uuid "$ESM_UUID" --enabled

The messages stay in the queue until their retention period expires, four days by default. For functions behind an API there is a harsher option: with a reserved concurrency of 0, Lambda throttles every invocation and callers get errors back.

aws lambda put-function-concurrency --function-name konzertkasse-prod-issueTickets --reserved-concurrent-executions 0

# Rollback
aws lambda delete-function-concurrency --function-name konzertkasse-prod-issueTickets

Post in the incident channel that the brake has been pulled and who will release it. A forgotten emergency brake is a small incident of its own.

Add the runbook check to your pipeline

Rules like “no alarm without a runbook” only survive day-to-day work if a machine checks them. The following script reads the CloudFormation template that serverless package generates and checks every alarm: is there a runbook link, does the file exist in the repository, does it have all required sections, and was it tested within the last 180 days?

// scripts/check-runbooks.ts
import { readFileSync, existsSync } from 'node:fs';
import { basename, join } from 'node:path';

const ALARM_TYPES = new Set(['AWS::CloudWatch::Alarm', 'AWS::CloudWatch::CompositeAlarm']);
const REQUIRED_SECTIONS = ['## Goal', '## Diagnosis', '## Actions', '## Verify', '## Escalation'];
const MAX_DESCRIPTION = 1024; // CloudWatch limit for AlarmDescription

interface Resource {
  Type: string;
  Properties?: { AlarmDescription?: unknown };
}

export interface Template {
  Resources?: Record<string, Resource>;
}

export interface Options {
  readRunbook: (fileName: string) => string | undefined;
  today: Date;
  maxAgeDays: number;
}

function daysBetween(from: Date, to: Date): number {
  return Math.floor((to.getTime() - from.getTime()) / 86_400_000);
}

export function checkRunbook(fileName: string, content: string, options: Options): string[] {
  const problems: string[] = [];
  for (const section of REQUIRED_SECTIONS) {
    if (!content.includes(`\n${section}`)) {
      problems.push(`${fileName}: section "${section}" is missing`);
    }
  }
  const tested = /^Last tested: (\d{4}-\d{2}-\d{2})$/m.exec(content);
  if (!tested) {
    problems.push(`${fileName}: line "Last tested: YYYY-MM-DD" is missing`);
  } else {
    const age = daysBetween(new Date(`${tested[1]}T00:00:00Z`), options.today);
    if (age > options.maxAgeDays) {
      problems.push(`${fileName}: last tested ${age} days ago (allowed: ${options.maxAgeDays})`);
    }
  }
  return problems;
}

export function checkTemplate(template: Template, options: Options): string[] {
  const problems: string[] = [];
  const checked = new Set<string>();

  for (const [logicalId, resource] of Object.entries(template.Resources ?? {})) {
    if (!ALARM_TYPES.has(resource.Type)) continue;

    const description = resource.Properties?.AlarmDescription;
    if (typeof description !== 'string') {
      problems.push(`${logicalId}: AlarmDescription is missing or not a plain string`);
      continue;
    }
    if (description.length > MAX_DESCRIPTION) {
      problems.push(`${logicalId}: AlarmDescription has ${description.length} characters (max. ${MAX_DESCRIPTION})`);
    }

    const link = /Runbook: (https:\/\/\S+\.md)/.exec(description);
    if (!link) {
      problems.push(`${logicalId}: no runbook link in the alarm text`);
      continue;
    }

    const fileName = basename(link[1]);
    const content = options.readRunbook(fileName);
    if (content === undefined) {
      problems.push(`${logicalId}: runbook ${fileName} does not exist in the repository`);
      continue;
    }
    if (!checked.has(fileName)) {
      checked.add(fileName);
      problems.push(...checkRunbook(fileName, content, options));
    }
  }
  return problems;
}

function main(): void {
  const templatePath = process.argv[2] ?? '.serverless/cloudformation-template-update-stack.json';
  const runbookDir = process.argv[3] ?? 'docs/runbooks';
  const template = JSON.parse(readFileSync(templatePath, 'utf8')) as Template;

  const problems = checkTemplate(template, {
    readRunbook: (fileName) => {
      const path = join(runbookDir, fileName);
      return existsSync(path) ? readFileSync(path, 'utf8') : undefined;
    },
    today: new Date(),
    maxAgeDays: 180,
  });

  if (problems.length > 0) {
    console.error(`Runbook check failed:\n- ${problems.join('\n- ')}`);
    process.exit(1);
  }
  console.log('Runbook check: every alarm links to a maintained runbook.');
}

if (process.argv[1]?.endsWith('check-runbooks.ts')) {
  main();
}

The tests cover the key cases, from a missing link to an outdated test date. They run with the built-in Node.js test runner:

// test/check-runbooks.test.ts
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { checkTemplate, type Template } from '../scripts/check-runbooks.ts';

const goodRunbook = [
  '# RB-01: Orders in the DLQ',
  'Last tested: 2026-11-02',
  '## Goal', '## Diagnosis', '## Actions', '## Verify', '## Escalation',
].join('\n');

function alarm(description: unknown): Template {
  return {
    Resources: {
      DlqAlarm: { Type: 'AWS::CloudWatch::Alarm', Properties: { AlarmDescription: description } },
      Queue: { Type: 'AWS::SQS::Queue' },
    },
  };
}

const options = {
  readRunbook: (name: string) => (name === 'rb-01.md' ? goodRunbook : undefined),
  today: new Date('2026-11-07T00:00:00Z'),
  maxAgeDays: 180,
};

test('accepts an alarm with a maintained runbook', () => {
  const problems = checkTemplate(alarm('DLQ full. Runbook: https://example.com/runbooks/rb-01.md'), options);
  assert.deepEqual(problems, []);
});

test('reports an alarm without a runbook link', () => {
  const problems = checkTemplate(alarm('DLQ full.'), options);
  assert.match(problems[0], /no runbook link/);
});

test('reports a runbook missing from the repository', () => {
  const problems = checkTemplate(alarm('Runbook: https://example.com/runbooks/rb-99.md'), options);
  assert.match(problems[0], /rb-99\.md does not exist/);
});

test('reports an outdated runbook', () => {
  const later = { ...options, today: new Date('2027-08-01T00:00:00Z') };
  const problems = checkTemplate(alarm('Runbook: https://example.com/runbooks/rb-01.md'), later);
  assert.match(problems[0], /last tested \d+ days ago/);
});

test('reports missing required sections', () => {
  const thin = { ...options, readRunbook: () => 'Last tested: 2026-11-01\n## Goal' };
  const problems = checkTemplate(alarm('Runbook: https://example.com/runbooks/rb-01.md'), thin);
  assert.equal(problems.length, 4);
});

The package.json ties everything together. Note that serverless package does not need access to your AWS account as long as you do not resolve variables from Parameter Store or Secrets Manager:

{
  "name": "runbook-lambda",
  "version": "1.0.0",
  "description": "Runbooks for a Lambda service plus a CI check that every alarm links to one",
  "scripts": {
    "test": "tsx --test test/*.test.ts",
    "typecheck": "tsc --noEmit",
    "check:runbooks": "serverless package --stage prod && tsx scripts/check-runbooks.ts"
  },
  "license": "MIT",
  "devDependencies": {
    "@types/aws-lambda": "^8.10.164",
    "@types/node": "^26.6.6",
    "esbuild": "^0.28.2",
    "serverless": "^3.40.0",
    "serverless-esbuild": "^1.57.2",
    "tsx": "^4.23.15",
    "typescript": "^7.0.2"
  },
  "type": "module"
}

npm install
npm test
npm run check:runbooks

If everything is in order, the check reports Runbook check: every alarm links to a maintained runbook. If a file is missing, it exits with code 1 and the pipeline stays red:

Runbook check failed:
- IssueTicketsErrorsAlarm: runbook rb-02-rollback-issue-tickets.md does not exist in the repository

In GitHub Actions, one extra step run: npm run check:runbooks after the tests is enough. The 180 days are my own assumption. Pick a period in which you can realistically walk through every runbook once.

Keeping runbooks up to date

A runbook gets a little more outdated with every deploy. The Well-Architected Framework names exactly that as a typical problem: runbooks that no longer match the system. Three habits help. First, every runbook gets an owner, most easily through a line for docs/runbooks/ in the CODEOWNERS file. Second, every incident ends with the question of what was missing in the runbook. In the blameless postmortem, that turns into a concrete action item. Third, you walk through runbooks regularly so that “Last tested” does not become a decorative field.

And when should you automate? I would only automate once a runbook has run several times without changes, for example as a Systems Manager Automation runbook or as a script in the repository. Before that, you are automating a procedure the team does not fully understand yet and just hiding the uncertainty in code.

Common mistakes when you write a runbook

• One document for everything: nobody reads a 40-page operations manual at night. One alarm, one runbook, one screen.
• Placeholders instead of commands: “insert queue URL” costs minutes at 3 a.m. Set variables once at the top, then just copy.
• No verification step: without a measurable end, “looks good” becomes the criterion.
• Permissions not checked: if you write a runbook and test it with admin rights, it is easy to miss the key point. The on-call role may not even be allowed to start the redrive.
• Sensitive data in the text: credentials, customer data or internal passwords have no place in a runbook.

My take

Being asked to write a runbook often feels like a chore. To me, runbooks are less documentation than care. They decide whether an incident at night is half an hour of routine or an hour of panic with three phone calls. That is why I hold a clear position: an alarm without a runbook is an unfinished feature and does not belong in the on-call rotation.

Where I would push back is the reflex to demand a runbook for everything. A runbook requirement for every little thing quickly turns into bureaucracy, and then people write texts nobody reads. I would limit runbooks to two cases: alarms that wake people up, and risky manual tasks that rarely happen. An AI assistant can help when you write a runbook, for example with structuring it or phrasing the if-then cases. I would still run every single command in it myself in staging before it goes into the runbook.

FAQ

What is the difference between a runbook and a playbook?

A runbook leads to a defined outcome, such as “DLQ empty, alarm OK”. A playbook helps you investigate a problem while the cause is still unclear. The runbook often follows the playbook.

Where should runbooks be stored?

Ideally as Markdown in the same repository as the code, with an owner set via CODEOWNERS. That way code and runbook change in the same pull request. A wiki works too if someone is responsible and the link is in the alarm.

How long should a runbook be?

Short enough to fit on one screen. If it gets longer, split it into several runbooks, for example one per alarm.

How do I start to write a runbook?

Take the alarm that wakes people up most often and write down the goal and impact first. Then add diagnosis commands, actions as if-then cases, a verification step and the escalation.

Can I automate runbooks?

Yes, for example with AWS Systems Manager Automation or with scripts in the repository. It makes sense once a runbook has worked several times without changes.

Your next step

Take the alarm that last woke you or your team at night and set aside one evening to write a runbook for it using the template from this article. Then have someone on the team walk through it in staging and enter the date. I am curious: which section is missing most often in your runbooks, verify, rollback or escalation?

Sources

• AWS Well-Architected Framework: OPS07-BP03 Use runbooks to perform procedures
• AWS Well-Architected Framework: OPS07-BP04 Use playbooks to investigate issues
• Google SRE Book: Introduction
• Amazon SQS API Reference: StartMessageMoveTask
• Amazon SQS: Troubleshoot dead-letter queue and DLQ redrive issues
• AWS Lambda: Creating and configuring an Amazon SQS event source mapping
• AWS Lambda: Configuring reserved concurrency
• Amazon CloudWatch API Reference: PutMetricAlarm
• AWS Systems Manager Automation
• Serverless Framework: rollback
• Amazon CloudWatch Pricing

This post was created with AI assistance.

👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en