This document captures emergency actions and immediate procedures when infrastructure failures threaten service availability or data integrity.
- Confirm the alert is valid and not a false positive.
- Identify the impacted system(s) and scope.
- Determine if the incident is an emergency requiring immediate escalation.
- Assign a lead and open the incident communication channel.
- Symptoms: inability to reach API endpoints, failed health checks, DNS resolution errors.
- Immediate actions:
- Confirm DNS record status with DNS provider.
- Check network ACLs, firewall rules, and load balancer health.
- Fail over to secondary ingress if configured.
- Escalate to networking specialist and provider support.
- Symptoms: control plane unreachable, nodes not ready, persistent pod failures.
- Immediate actions:
- Verify cluster control plane health in cloud provider console.
- Check node status with
kubectl get nodes. - If nodes are unhealthy, do not delete pods blindly; assess whether scaling or reboot is safer.
- Restart failed system pods and inspect kubelet logs.
- Escalate to cluster operations and provider support.
- Symptoms: failed queries, connection errors, replication lag, data inconsistencies.
- Immediate actions:
- Switch read/write traffic to standby or replica nodes if available.
- Review recent schema or configuration changes.
- Restore from the most recent validated backup only when restore path is verified.
- Escalate to database owner and DBA support.
- Symptoms: unexpected privilege changes, unknown infrastructure access, data exfiltration alerts.
- Immediate actions:
- Isolate affected systems and revoke compromised credentials.
- Disable access to public endpoints if needed.
- Notify Security Incident Response immediately.
- Preserve logs and evidence for investigation.
- Incident channel created and active
- Incident leader assigned
- On-call engineer notified
- Critical service impact confirmed
- Traffic isolation or failover initiated if needed
- External provider support engaged if needed
- Customer impact and status page updated
- Keep provider ticket number and contact details in the incident notes.
- When opening support cases, provide:
- Incident summary
- Affected resources and time window
- Steps already taken
- Desired recovery target
- Track communications and update the incident timeline.
- Ensure service endpoints pass health checks.
- Run synthetic smoke tests for impacted workflows.
- Confirm metrics and alerts return to normal thresholds.
- Validate that the recovered state matches expected production behavior.
- Document the final state and any temporary mitigations.
- Schedule a follow-up review with the incident response team.
- Create or update the post-mortem.