Setting Up Alerting with PagerDuty
= Setting Up Alerting with PagerDuty =
This guide outlines the process of integrating your server's monitoring systems with PagerDuty for robust alerting, on-call management, and incident response. PagerDuty is a leading digital operations management platform that helps teams respond to critical events quickly and efficiently. By setting up PagerDuty, you ensure that the right people are notified when something goes wrong on your server, minimizing downtime and its impact.
Prerequisites
Before you begin, ensure you have the following:
- A PagerDuty account. If you don't have one, you can sign up for a free trial at PagerDuty.com.
- A PagerDuty service created and configured for your server. This involves defining how alerts will be received (e.g., via email, API).
- Basic familiarity with your server's operating system (e.g., Linux).
- Root or sudo privileges on your server.
- A monitoring tool or script configured to send alerts. This could be Nagios, Zabbix, Prometheus Alertmanager, or even a custom script.
- **Alerts not arriving in PagerDuty:** * **Check Integration Key:** Ensure the integration key in your monitoring tool or script exactly matches the one in PagerDuty. Even a single character difference will cause failure. * **Check PagerDuty URL:** Verify that the PagerDuty Events API URL (`https://events.pagerduty.com/v2/enqueue`) is correct and accessible from your server. * **Firewall Issues:** Ensure no firewalls on your server or network are blocking outbound connections to PagerDuty's API endpoints (typically on port 443). * **JSON Payload Format:** Incorrect JSON formatting is a common culprit. Use a JSON validator to check your payload. Ensure all required fields (`routing_key`, `event_action`, `payload.summary`, `payload.source`) are present. * **PagerDuty Service Status:** Confirm that the PagerDuty service itself is active and not disabled.
- **Alerts are triggered but not notifying the correct person:** * **Escalation Policy:** Review your PagerDuty Escalation Policy. Is it correctly configured with the right users and escalation rules? * **Service Assignment:** Ensure the PagerDuty service is assigned to the correct Escalation Policy. * **On-Call Schedule:** Verify that the on-call schedule for the relevant Escalation Policy is up-to-date.
- **Alerts are not resolving:** * **"Resolve" Event Not Sent:** Your monitoring system or script needs to explicitly send a "resolve" event to PagerDuty when the condition is no longer met. The `event_action` should be `"resolve"`, and you typically need to include a `dedup_key` that matches the original trigger event. * **Incorrect `dedup_key`:** If using `dedup_key` for resolution, ensure it matches the `dedup_key` (or a combination of fields that PagerDuty uses for de-duplication) of the original trigger event.
- **Event Rules:** PagerDuty allows you to create advanced event rules to suppress, route, or transform incoming events based on various criteria (e.g., severity, component, source).
- **Custom Event Fields:** Utilize custom fields in your PagerDuty events to pass richer context to your responders, aiding in faster diagnosis.
- **Integrations with other tools:** PagerDuty integrates with a vast array of tools, including Slack, Jira, and various cloud providers, to streamline your incident response workflow.
Understanding PagerDuty Integration
PagerDuty integrates with your monitoring tools through various "Integrations." For most server-level monitoring, you'll likely use an "Events API v2" integration or an email integration. The Events API v2 is generally preferred due to its flexibility and richer data capabilities.
When an event (alert) is triggered by your monitoring system and sent to PagerDuty, PagerDuty evaluates the event against your service's rules. If the event meets the criteria for an incident, PagerDuty will notify the on-call person or escalation policy defined for that service.
Step 1: Creating a PagerDuty Service and Integration
1. **Log in to your PagerDuty account.** 2. Navigate to the **Services** tab and click **New Service**. 3. Give your service a descriptive name, such as "MyWebServer Monitoring." 4. Select an **Escalation Policy**. If you don't have one, you'll need to create one first, defining who gets notified and in what order. 5. Under **Integrations**, click **Add Integration**. 6. Search for and select your monitoring tool. If you're using a generic approach or custom scripts, choose **Events API v2**. 7. Click **Add Integration**. 8. **Note the Integration Key** provided. This key is crucial for authenticating your alerts. It will look something like `a1b2c3d4e5f678901234567890abcdef`.
Step 2: Configuring Your Monitoring Tool to Send Alerts to PagerDuty
The exact method for sending alerts depends on your monitoring system. Here are examples for common scenarios:
Using `curl` with Events API v2 (for custom scripts or simple checks)
This is a versatile method that can be used from any shell script.
1. **Create a simple alert script** (e.g., `/opt/scripts/check_service.sh`):
```bash #
# Replace with your actual PagerDuty Integration Key PAGERDUTY_INTEGRATION_KEY="YOUR_PAGERDUTY_INTEGRATION_KEY" # Replace with your PagerDuty Events API URL PAGERDUTY_URL="https://events.pagerduty.com/v2/enqueue"
# Example: Check if a web server is responding if
# Construct the JSON payload
JSON_PAYLOAD=$(cat < # Send the alert to PagerDuty
curl -X POST \
-H "Content-Type: application/json" \
--data "$JSON_PAYLOAD" \
"$PAGERDUTY_URL" echo "Alert sent to PagerDuty: $EVENT_SUMMARY"
else
echo "Web server on localhost is running."
# You can optionally send a 'resolve' event here if the service was previously down
# See PagerDuty documentation for 'resolve' event payload structure.
fi
```
2. **Make the script executable:**
```bash
chmod +x /opt/scripts/check_service.sh
```
3. **Test the script:**
```bash
/opt/scripts/check_service.sh
```
If the web server is down, you should see output indicating an alert was sent. Check your PagerDuty dashboard for an incident. Security Implication: Never hardcode sensitive credentials directly in scripts that are world-readable. For integration keys, consider using environment variables or a secure configuration file with restricted permissions. Performance Note: `curl` is generally efficient for sending small JSON payloads. For high-volume alerting, consider more optimized methods or batching if your monitoring system supports it.Using Prometheus Alertmanager
If you are using Prometheus for metrics collection, Alertmanager is its native alerting component.
1. **Configure Alertmanager's `alertmanager.yml`:** Add a PagerDuty receiver to your `alertmanager.yml` configuration file.
```yaml route: group_by: ['alertname', 'cluster'] group_wait: 30s group_interval: 5m repeat_interval: 1h receiver: 'default-receiver' # Fallback receiver
receivers: - name: 'default-receiver' pagerduty: - service_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY' # Optional: You can customize the event payload here # client: 'Prometheus' # client_url: 'http://your-prometheus-server.com' # description: '' # severity: '' ``` 2. **Reload Alertmanager configuration:** The method depends on how you run Alertmanager (e.g., systemd, Docker). For systemd: ```bash sudo systemctl reload alertmanager ``` 3. **Configure Prometheus to send alerts to Alertmanager:** In your `prometheus.yml` file, ensure the `alerting` section points to your Alertmanager instance.
```yaml alerting: alertmanagers: - static_configs: - targets: ['localhost:9093'] # Replace with your Alertmanager address ``` 4. **Define Alerting Rules in Prometheus:** Create or edit rule files (e.g., `rules.yml`) to define what conditions trigger alerts.
```yaml groups: - name: general.rules rules: - alert: HighRequestLatency expr: job:request_latency_seconds:mean5m{job="my-app"} > 0.5 for: 5m labels: severity: critical annotations: summary: "High request latency detected on " description: " has a request latency above 0.5s for more than 5 minutes." ``` Prometheus will evaluate these rules. If a rule fires and persists for the `for` duration, it will send the alert to Alertmanager, which then forwards it to PagerDuty.
Performance Note: Prometheus and Alertmanager are highly efficient for collecting and processing metrics and alerts at scale. The integration relies on Alertmanager's robust routing and notification capabilities.
Step 3: Testing Your PagerDuty Integration
1. **Trigger an alert:** If you used the `curl` example, intentionally stop the service being monitored (e.g., `sudo systemctl stop apache2`). Then run your script again. 2. **Check PagerDuty:** Log in to your PagerDuty dashboard. You should see a new incident triggered for your service. 3. **Acknowledge the incident:** In PagerDuty, acknowledge the incident to signal that you are working on it. This stops further escalations for that incident. 4. **Resolve the incident:** Once the issue is fixed (e.g., `sudo systemctl start apache2`), trigger your script again, or if your monitoring system automatically detects the resolution, it should send a "resolve" event to PagerDuty. The incident should then be marked as resolved.
Troubleshooting Common Issues
Advanced Features
By implementing PagerDuty, you move beyond simple notifications to a structured incident management process, significantly improving your team's ability to handle critical server events.
Category:Monitoring Category:Alerting Category:PagerDuty Category:Linux Administration