Monitoring & Alerting Setup Prompt (ChatGPT)

## System
You are an expert AI assistant. Respond using clear markdown formatting.

## User
You are a senior site reliability engineer specialising in observability.

Design a monitoring and alerting strategy for the following system:

System description: {{system_description}}
Tech stack: {{tech_stack}}
Monitoring tools: {{monitoring_tools}}
SLA requirements: {{sla}}
On-call setup: {{on_call}}

Provide:
1. **Key metrics to track** — list the top 10 metrics with thresholds for warning and critical alerts
2. **SLO definitions** — availability, latency, and error rate SLOs based on the stated SLA
3. **Alert rules** — for each critical metric: condition, severity, and action
4. **Dashboard panels** — describe 6-8 key panels for an operations dashboard
5. **Runbook outline** — for the top 3 most likely alerts, a short runbook with diagnosis steps
6. **Alert noise reduction** — strategies to avoid alert fatigue (grouping, deduplication, inhibition rules)

Variables

Example

system_description: REST API for e-commerce checkout, processes credit card payments
tech_stack: Node.js, PostgreSQL, Stripe API, Redis, deployed on Kubernetes
monitoring_tools: Prometheus, Grafana, PagerDuty
sla: 99.9% uptime, p99 latency < 1 second
on_call: 24/7 rotation, target < 3 critical pages per week

**Key Metrics:**
1. HTTP error rate (5xx): warn >1%, critical >5%
2. p99 response time: warn >800ms, critical >1500ms
3. Stripe API error rate: warn >2%, critical >10%
4. Database connection pool utilisation: warn >70%, critical >90%
5. Pod restart rate: warn >2/hour, critical >10/hour

**SLO Definitions:**
- Availability: 99.9% of requests return non-5xx within 30 days
- Latency: 95% of checkout requests complete within 1 second

Related Tools