Broadcasting prometheus alerts to webex space.

In this guide, we will walk you through integrating Prometheus with Webex to send alerts directly to your Webex spaces. This is especially useful for teams using Webex for collaboration, as it allows you to receive real-time alerts on the platform.

Overview

We’ll cover the following:

• Creating a Webex Bot.
• Retrieving the room_id using the Webex API.
• Configuring Alertmanager to send alerts to Webex.
• Securing your Bot Token with Kubernetes secrets.

1. Prerequisites:
To get started, ensure that you have Prometheus and Alertmanager installed. If not, you can refer to the Prometheus documentation for installation instructions.

Tools required:

1. Prometheus and Alertmanager.
2. Webex account with developer access.
3. Working Kubernetes cluster (for deploying Alertmanager).

2. Creating a Webex Bot:

To integrate Webex with Prometheus, you'll first need to create a bot in Webex Teams that will handle receiving and sending messages on
your behalf.

Steps:

1. Log in to the Webex Developer Portal: Head to the Webex Developer Portal and log in.
2. Navigate to My Apps: Once logged in, click on "My Apps" from the top navigation bar.
3. Create a New App: Select "Create a New App" and choose Create a Bot.

Fill in the Details:

1. Bot Name: Choose a name that reflects the bot's purpose (e.g., PrometheusAlertBot).
2. Bot Username: A unique identifier for your bot.
3. Generate Access Token: After the bot is created, you’ll be provided with an access_token. Copy this token and save it securely. This will be used later to authenticate API requests.

3. Invite the Bot to a Webex Channel:
Now that you have a bot, you need to add it to a Webex space where it can post alerts.

Steps:

1. Create a Webex Space: In Webex Teams, create a space (channel) for your alerts.
2. Invite the Bot: Go to the "Add People" section of your space and invite your bot by using its bot username.
3. Once added, your bot will be able to receive and send messages within the space.

4. Retrieving the room_id Using the Webex API:

To configure Prometheus to send alerts to Webex, you'll need the room_id of the space where you invited your bot. Steps to Retrieve room_id:
Open your terminal and run the following command, replacing YOUR_ACCESS_TOKEN with the token you generated earlier:

  curl -s -X GET -H 'Authorization: Bearer YOUR_ACCESS_TOKEN' https://webexapis.com/v1/rooms | jq .

The response will contain a list of rooms the bot is part of. Look for the room where you invited the bot and copy the room_id.
This room_id will be used to configure Alertmanager to send alerts to the right Webex space.

5. Configuring Alertmanager to Send Alerts to Webex
Now that you have the room_id and access_token, it's time to configure Alertmanager to forward alerts to Webex.

Step 1: Modify alertmanager.yaml
Update your Alertmanager configuration to include a new receiver for Webex:

receivers:
    - name: webexroom
      webex_configs:
        - room_id: "add_room_id_here"
          message: "CPU usage is high"
          http_config:
            authorization:
              type: Bearer
              credentials: "add_bot_token_here"

In this configuration:

Replace add_room_id_here with the actual room_id you retrieved earlier.
Replace add_bot_token_here with the access token generated for your bot.

Step 2: Secure Your Bot Token Using Kubernetes Secrets
Since the bot token is sensitive information, it’s a best practice to store it securely using Kubernetes secrets, create k8s Secret.
Save your bot token in base64 encoding (you can encode it using echo -n "your_token" | base64, and create the following secret
configuration file:

secret.yaml

  apiVersion: v1
  kind: Secret
  metadata:
    name: apc-webex-receiver
  type: Opaque
  data:
    Bearer: your_encoded_token

kubectl -n monitoring apply -f secret.yaml

Add the secret to AlertmanagerSpec, In your Alertmanager Helm chart or Kubernetes configuration, add the secret name to the

alertmanagerSpec:

  alertmanagerSpec:
    secrets:
    - apc-webex-receiver

Modify Alertmanager Config to Use Secret:
Update the credentials_file to point to the secret in your Kubernetes cluster:

receivers:
    - name: webexroom
      webex_configs:
        - room_id: "your_room_id"
          message: "CPU usage is high"
          http_config:
            authorization:
              type: Bearer
              credentials_file: /etc/alertmanager/secrets/apc-webex-receiver/Bearer

This ensures your token is securely managed and not hardcoded into configuration files.

6. Using Templating for Dynamic Alerts:
If you want to loop through multiple alerts and send them as a formatted message, you can use the templating feature in Alertmanager’s message field.

For example:

 message: >-
     {{ range .Alerts }}
       *Alert:* {{ .Labels.alertname }}
       *Cluster:* {{ .Labels.cluster }}
     {{ end }}

This will format the alerts in a user-friendly way, providing more context in your Webex notifications.