Alerting System
Alerting System Server Configuration
This article details the server configuration for the Alerting System, a crucial component of our infrastructure monitoring suite. It is aimed at new system administrators and engineers who will be responsible for its maintenance and operation. The Alerting System is responsible for receiving alerts from various monitoring sources (like Nagios, Zabbix, and custom scripts) and notifying the appropriate personnel via various channels (email, PagerDuty, Slack).
Overview
The Alerting System comprises three primary servers: a receiver, a processor, and a notifier. Each server has a specific role in handling alerts. The receiver accepts incoming alerts, the processor enriches and categorizes them, and the notifier delivers them to the designated recipients. This distributed architecture ensures high availability and scalability. Proper configuration of each component is vital for reliable alerting. The system heavily relies on RabbitMQ for message queuing and inter-server communication. Consider reviewing the RabbitMQ documentation if you encounter issues with message delivery.
Server Specifications
Below are the specifications for each server within the Alerting System. These specifications are based on current load and projected growth.
| Server Role | Hostname | CPU | Memory (RAM) | Storage (Disk) | Operating System |
|---|---|---|---|---|---|
| Receiver | alert-receiver-01.example.com | 8 Cores | 16 GB | 250 GB SSD | Ubuntu Server 22.04 LTS |
| Processor | alert-processor-01.example.com | 12 Cores | 32 GB | 500 GB SSD | CentOS Stream 9 |
| Notifier | alert-notifier-01.example.com | 4 Cores | 8 GB | 100 GB SSD | Debian 11 |
Software Stack
Each server utilizes a specific software stack to perform its functions. The following table outlines the core software components installed on each server. Note the version numbers are current as of October 26, 2023, and may require updates. Always check the Software Repository for the latest versions.
| Server Role | Software | Version | ||||||
|---|---|---|---|---|---|---|---|---|
| Receiver | Nginx | 1.23.3 | | Python 3 | 3.10.6 | | Alerting Receiver Script | v1.2.0 | ||
| Processor | RabbitMQ | 3.9.13 | | Redis | 7.0.5 | | Python 3 | 3.10.6 | | Alerting Processor Script | v1.3.1 |
| Notifier | RabbitMQ | 3.9.13 | | Python 3 | 3.10.6 | | Alerting Notifier Script | v1.1.0 |
Configuration Details
The configuration files for each server are managed via Ansible for consistency and ease of deployment. Key configuration aspects include:
- Receiver: The Nginx configuration is optimized for handling a high volume of incoming POST requests. The Python script listens on port 8080 and validates incoming alert payloads against a predefined schema. Any invalid alerts are logged to syslog.
- Processor: The RabbitMQ server is configured with multiple queues for different alert severities (Critical, Warning, Info). The Python script consumes messages from these queues and performs enrichment, deduplication, and categorization. It utilizes Redis for caching frequently accessed data. Consult the Redis Configuration Guide for optimization tips.
- Notifier: The RabbitMQ server is configured as a consumer for the processed alert queues. The Python script formats the alerts and sends them to the appropriate notification channels. Configuration for each channel (email, PagerDuty, Slack) is stored in a dedicated configuration file, protected by file permissions.
- Firewall: A restrictive firewall policy is enforced on each server.
- TLS/SSL: Communication between servers and external services is encrypted using TLS/SSL.
- Authentication: Access to the RabbitMQ management interface is protected by strong passwords and restricted to authorized users. Refer to the Access Control List.
- Regular Security Audits: Regular security audits are conducted to identify and address potential vulnerabilities. See the Security Audit Schedule.
- Log Monitoring: All server logs are centrally collected and monitored for suspicious activity using ELK Stack.
- Alerts not being received: Check the RabbitMQ queues for backlog. Verify the Receiver script is running and accepting connections.
- Alerts not being processed: Check the Processor script logs for errors. Ensure Redis is running and accessible.
- Notifications not being sent: Check the Notifier script logs for errors. Verify the configuration for the notification channel is correct.
- Telegram: @powervps Servers at a discounted price
Network Configuration
Effective network configuration is paramount for the Alerting System's performance. Important considerations include:
| Parameter | Value | ||||||
|---|---|---|---|---|---|---|---|
| Firewall Rules | Allow inbound traffic on port 8080 (Receiver) | | Allow outbound traffic to RabbitMQ (all servers) | | Allow outbound traffic to notification services (Notifier) | | DNS Resolution | Ensure proper DNS resolution for all servers and external services | | Network Monitoring | Monitor network latency and bandwidth usage between servers |
Security Considerations
Security is a top priority. The following measures are in place:
Troubleshooting
Common issues and their potential solutions:
Refer to the Troubleshooting Guide for more detailed troubleshooting steps. Also, consult the FAQ for commonly asked questions.
Intel-Based Server Configurations
| Configuration | Specifications | Benchmark |
|---|---|---|
| Core i7-6700K/7700 Server | 64 GB DDR4, NVMe SSD 2 x 512 GB | CPU Benchmark: 8046 |
| Core i7-8700 Server | 64 GB DDR4, NVMe SSD 2x1 TB | CPU Benchmark: 13124 |
| Core i9-9900K Server | 128 GB DDR4, NVMe SSD 2 x 1 TB | CPU Benchmark: 49969 |
| Core i9-13900 Server (64GB) | 64 GB RAM, 2x2 TB NVMe SSD | |
| Core i9-13900 Server (128GB) | 128 GB RAM, 2x2 TB NVMe SSD | |
| Core i5-13500 Server (64GB) | 64 GB RAM, 2x500 GB NVMe SSD | |
| Core i5-13500 Server (128GB) | 128 GB RAM, 2x500 GB NVMe SSD | |
| Core i5-13500 Workstation | 64 GB DDR5 RAM, 2 NVMe SSD, NVIDIA RTX 4000 |
AMD-Based Server Configurations
| Configuration | Specifications | Benchmark |
|---|---|---|
| Ryzen 5 3600 Server | 64 GB RAM, 2x480 GB NVMe | CPU Benchmark: 17849 |
| Ryzen 7 7700 Server | 64 GB DDR5 RAM, 2x1 TB NVMe | CPU Benchmark: 35224 |
| Ryzen 9 5950X Server | 128 GB RAM, 2x4 TB NVMe | CPU Benchmark: 46045 |
| Ryzen 9 7950X Server | 128 GB DDR5 ECC, 2x2 TB NVMe | CPU Benchmark: 63561 |
| EPYC 7502P Server (128GB/1TB) | 128 GB RAM, 1 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (128GB/2TB) | 128 GB RAM, 2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (128GB/4TB) | 128 GB RAM, 2x2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (256GB/1TB) | 256 GB RAM, 1 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (256GB/4TB) | 256 GB RAM, 2x2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 9454P Server | 256 GB RAM, 2x2 TB NVMe |
Order Your Dedicated Server
Configure and order your ideal server configurationNeed Assistance?
⚠️ *Note: All benchmark scores are approximate and may vary based on configuration. Server availability subject to stock.* ⚠️