CD ChatOps with Slack & Buildkite

Buildkite is my preferred deployment pipeline system. I prefer Buildkite because I can run agents on my own infrastructure. This means I can control AWS access with IAM and even bake AMIs with all dependencies for faster pipelines.

Buildkite pipelines may be triggered from GitHub deployments. The only catch is making a nice UI for triggering deployments. I recently started using SlashDeploy (same name, but no affiliation) to trigger deploys ChatOps style from Slack. Here’s how it works.

ChatOps via Slack

SlashDeploy adds the /deploy command to Slack. /deploy is useful because it is a small and sharp tool. It only triggers GitHub deployments. This a clear integration point with other systems. This means /deploy can integrate with any deployment pipeline. /deploy and BuildKite work especially well together because /deploy maps directly to Buildkite pipelines. /deploy can also specify the Github Deployment task (such as migrate, seed, or activate maintenance) which may processed inside Buildkite to trigger other pipelines. Plus, all this happens in Slack so anyone can /deploy APP or /deploy APP with TASK.

Building the Deployment Pipeline

Buildkite Pipeline Overview.

I prefer continuous deployment with options for triggering manual deployments when needed. /deploy supports both scenarios. /deploy is configured to trigger automatic deployments for my intended branch if tests pass in .slashdeploy.yml:

version: 1
environments:
  production:
    # Important to everyone see how to deploy
    respond_in_channel: true
    # For notifications
    channel: ops
    checks:
      - buildkite/mono-tests
    auto_deploy:
      # auto deploy master
      ref: refs/heads/master

I use a custom pipeline script to process the particular deployment environment and task. This is deployment pipeline’s entry point. Buildkite pipelines can trigger other pipelines as step in larger pipelines. Here’s an example.

The production deploy does not call the seed pipeline, but a dev deploy does. Team members can also invoke /deploy app with seed. The deployment task is set to seed in this case. The default value is deploy. My pipeline script checks these two values then loads the relevant pipeline file via buildkite pipeline upload. Here's a skeleton:

#!/usr/bin/env bash

set -euo pipefail

main() {
    local environment="${BUILDKITE_GITHUB_DEPLOYMENT_ENVIRONMENT?required}"
    local task="${BUILDKITE_GITHUB_DEPLOYMENT_TASK?required}"

    local pipeline=".buidlkite/${environment}-${task}.yml"

    if [ -f "${pipeline}" ]; then
        buildkite pipeline upload "${pipeline}"
    else
        echo "Cannot handle ${envrionment}/${task} invocation!" 1>&2
        return 1
    fi
}

main "$@"

This approach keeps it simple by mapping each environment/task to a specific pipeline file. The pipelines may also trigger other pipelines like so:

- label: ':rocket: :seedling:'
    trigger: mono-seed
    build:
      commit: "${BUILDKITE_COMMIT?}"
      branch: "${BUILDKITE_BRANCH?}"
      env:
        BUILDKITE_GITHUB_DEPLOYMENT_ENVIRONMENT: "${BUILDKITE_GITHUB_DEPLOYMENT_ENVIRONMENT?}"
        BUILDKITE_GITHUB_DEPLOYMENT_TASK: "${BUILDKITE_GITHUB_DEPLOYMENT_TASK?}"

Lastly, I used Buildkite annotations to decorate the pipeline UI with the environment and task. This information is hidden by a few clicks otherwise. It’s useful when scrolling pipeline views to find the relevant build. Here’s a screenshot of production deploy with an annotation.

Production deploy with annotation and a skipped step.

Adding the annotation requires adding an additional pipeline step with an associated script. Buildkite annotations are Markdown. It was easier to handle whitespace sensitive strings and environment variable substitution in a separate file. Here are the relevant code snippets:

# the pipeline step:
steps:
  - label: ':console: Annotate'
    command: script/buildkite/deploy-annotation | buildkite-agent annotate --style info

# The annotation script:
#!/usr/bin/env bash

set -euo pipefail

cat <<EOF
- Environment: **${BUILDKITE_GITHUB_DEPLOYMENT_ENVIRONMENT}**
- Task: **${BUILDKITE_GITHUB_DEPLOYMENT_TASK}**
EOF

Conclusion

Buildkite has long been my preferred deployment pipeline software. Now /deploy combined with dedicated deploy pipelines and ChatOps style GitHub deployment triggers make the setup is better than ever. If you're building a new deployment pipeline then I highly recommend Buildkite (and their Elastic Stack if running on AWS) paired with SlashDeploy for ChatOps and deploy triggers. It's easy and just works—a hard quality to find in software.