Prometheus

The kube-prometheus-stack Helm chart deploys Alertmanager with a default configuration that routes all alerts to a “null” receiver—effectively discarding them. This post documents configuring Alertmanager to send notifications to Slack. The Problem Default Alertmanager configuration: receivers: - name: "null" route: receiver: "null" # All alerts discarded Alerts fire, but nobody gets notified. Solution Architecture ┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────┐ │ Prometheus │────▶│ Alertmanager │────▶│ Slack │ │ (fires alerts) │ │ (routes & groups) │ │ (#alerts) │ └─────────────────────┘ └──────────────────────┘ └─────────────┘ │ ▼ ┌──────────────────────┐ │ Routing Rules │ ├──────────────────────┤ │ critical → 1h repeat │ │ warning → 4h repeat │ │ Watchdog → silenced │ └──────────────────────┘ Directory Structure monitoring/alertmanager/ ├── .env.example # Webhook URL template ├── .env # Actual webhook (gitignored) ├── create-secret.sh # Creates Kubernetes secret └── README.md # Setup documentation Setup Step 1: Create Slack Webhook Go to https://api.slack.com/apps Click “Create New App” → “From scratch” Name: Alertmanager, select your workspace Go to “Incoming Webhooks” → Toggle “Activate” Click “Add New Webhook to Workspace” Select the channel for alerts (e.g., #alerts) Copy the webhook URL Step 2: Create Kubernetes Secret # monitoring/alertmanager/.env.example SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ SLACK_CHANNEL=#alerts #!/bin/bash # monitoring/alertmanager/create-secret.sh set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" if [ -f "$SCRIPT_DIR/.env" ]; then source "$SCRIPT_DIR/.env" else echo "Error: .env file not found" exit 1 fi kubectl create secret generic alertmanager-slack-config \ --from-literal=slack-webhook-url="${SLACK_WEBHOOK_URL}" \ --from-literal=slack-channel="${SLACK_CHANNEL}" \ --namespace=monitoring \ --dry-run=client -o yaml | kubectl apply -f - Run the setup: ...