Complete observability stack for Nova Rewards platform with Prometheus, Grafana, and comprehensive alerting.
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Backend │────▶│ Prometheus │────▶│ Grafana │
│ (Metrics) │ │ (Scraping) │ │ (Dashboards)│
└─────────────┘ └──────┬───────┘ └─────────────┘
│
▼
┌──────────────┐
│ Alertmanager │
│ (Routing) │
└──────┬───────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌────────┐ ┌─────────┐ ┌──────────┐
│ Slack │ │PagerDuty│ │ Email │
└────────┘ └─────────┘ └──────────┘
- Port: 9090
- Purpose: Metrics collection and storage
- Retention: 30 days
- Scrape Interval: 15 seconds
- Port: 3000
- Purpose: Visualization and dashboards
- Default Credentials: admin/admin (change on first login)
- Port: 9093
- Purpose: Alert routing and notification
- Integrations: Slack, PagerDuty, Email
- Node Exporter (9100): System metrics (CPU, memory, disk)
- PostgreSQL Exporter (9187): Database metrics
- Redis Exporter (9121): Cache metrics
- Nginx Exporter (9113): Web server metrics
- Blackbox Exporter (9115): Endpoint health checks
cd monitoring
cp .env.example .env
# Edit .env with your configuration# Start all monitoring services
docker-compose -f docker-compose.monitoring.yml up -d
# Verify services are running
docker-compose -f docker-compose.monitoring.yml ps- Grafana: http://localhost:3000
- Prometheus: http://localhost:9090
- Alertmanager: http://localhost:9093
Edit alertmanager/alertmanager.yml with your notification channels:
- Slack webhook URL
- PagerDuty integration key
- Email SMTP settings
- HTTP request duration (histogram)
- Request rate by endpoint
- Error rate by status code
- Active requests
- CPU usage
- Memory usage
- Disk space
- Network I/O
- Connection count
- Query duration
- Transaction rate
- Cache hit ratio
- Database size
- Redis memory usage
- Cache hit/miss rate
- Eviction rate
- Connection count
- HighErrorRate: 5xx errors > 5% for 5 minutes
- ServiceDown: Service unreachable for 2 minutes
- PostgreSQLDown: Database unreachable for 2 minutes
- RedisDown: Cache unreachable for 2 minutes
- EndpointDown: Health check failing for 5 minutes
- HighLatency: 95th percentile > 1 second for 10 minutes
- HighCPUUsage: CPU > 80% for 10 minutes
- HighMemoryUsage: Memory > 85% for 10 minutes
- HighDatabaseConnections: Connections > 80% of max
- HighRedisMemory: Redis memory > 90%
- DiskSpaceLow: Disk space < 15%
- SSLCertificateExpiringSoon: Certificate expires in < 7 days
Detailed incident response procedures are available in the runbooks/ directory:
- Request rate and latency
- Error rate trends
- System resource usage
- Database and cache metrics
- CPU, memory, disk usage
- Network I/O
- Process metrics
- Query performance
- Connection pool status
- Slow queries
- Database size trends
- Redis memory usage
- Hit/miss ratio
- Eviction rate
- Command statistics
The monitoring stack complements existing CloudWatch monitoring:
- CloudWatch: Infrastructure-level metrics (ALB, EC2, RDS)
- Prometheus: Application-level metrics (API, business logic)
# In your main docker-compose.yml, add monitoring network
networks:
monitoring:
external: true
name: nova-rewards_monitoring
# Connect backend to monitoring network
services:
backend:
networks:
- default
- monitoring# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nova-backend
spec:
selector:
matchLabels:
app: nova-backend
endpoints:
- port: metrics
path: /metrics# Backup Prometheus data
docker run --rm -v nova-rewards_prometheus_data:/data -v $(pwd):/backup \
alpine tar czf /backup/prometheus-backup-$(date +%Y%m%d).tar.gz /data# Edit alert rules
vim prometheus/rules/alerts.yml
# Reload Prometheus configuration
curl -X POST http://localhost:9090/-/reload# Trigger test alert
curl -X POST http://localhost:9093/api/v1/alerts \
-H "Content-Type: application/json" \
-d '[{
"labels": {"alertname": "TestAlert", "severity": "warning"},
"annotations": {"summary": "Test alert"}
}]'# Check Prometheus targets
curl http://localhost:9090/api/v1/targets
# Check network connectivity
docker exec -it nova-prometheus wget -O- http://backend:4000/metrics# Verify Prometheus datasource
curl http://localhost:3000/api/datasources
# Test Prometheus query
curl -G 'http://localhost:9090/api/v1/query' \
--data-urlencode 'query=up'# Check alert rules
curl http://localhost:9090/api/v1/rules
# Check Alertmanager status
curl http://localhost:9093/api/v1/status
# View Alertmanager logs
docker logs nova-alertmanager- Set appropriate thresholds
- Use inhibition rules
- Group related alerts
- Set proper evaluation periods
- Create role-specific dashboards
- Use consistent naming conventions
- Add documentation panels
- Set appropriate time ranges
- Follow Prometheus naming conventions
- Use consistent labels
- Document custom metrics
- Avoid high-cardinality labels
- Limit scrape frequency for expensive metrics
- Use recording rules for complex queries
- Set appropriate retention periods
- Monitor Prometheus resource usage
# Enable Grafana authentication
GF_AUTH_ANONYMOUS_ENABLED=false
GF_AUTH_BASIC_ENABLED=true# Restrict Prometheus access
- "127.0.0.1:9090:9090" # Only localhost
# Use reverse proxy with authentication# Use environment variables for sensitive data
# Never commit credentials to git
# Rotate credentials regularlyFor issues or questions:
- Check runbooks in
runbooks/directory - Review Prometheus logs:
docker logs nova-prometheus - Review Grafana logs:
docker logs nova-grafana - Contact DevOps team: [Contact info]