Live
#16 high infra_monitor
Memory alert on claw-gateway1 — 95.0% (threshold: 92%)
Host: claw-gateway1 CAUSE: Memory exhaustion likely caused by accumulated processes or memory leak, compounded by high CPU utilization indicating resource contention. IMPACT: System performance degradation, potential application crashes, and risk of out-of-memory errors affecting service availability. ACTION: Immediately review and terminate non-essential processes, check for memory leaks, and consider restarting the system if memory cannot be freed within 15 minutes. CPU: 99.0% | Memory: 95.0%
Opened 2026-07-26 16:34 UTC · Resolved 2026-07-26 16:50 UTC
Handoff Notes ← Dashboard
Timeline
⬡
WEBHOOK
2026-07-26 16:34 UTC
Alert received from AI Infra Monitor. Host: claw-gateway1, Severity: HIGH
◎
CONTEXT AGGREGATED
2026-07-26 16:34 UTC
Sources available: 2/3 — Runbook: ✗ | Past incidents: ✓ | Infra health: ✓
✦
Response Plan
2026-07-26 16:34 UTC

Severity

HIGH – claw-gateway1 memory exhausted (95%), CPU pinned at 99%; risk of service crash and cascading failures.

Root Cause

  • Memory leak or runaway process accumulation in gateway service
  • High CPU contention preventing garbage collection/memory reclaim

Actions

  1. SSH to claw-gateway1; run ps aux --sort=-%mem and kill non-essential processes or memory hogs immediately.
  2. Check application logs for OOM errors or exceptions; restart claw-gateway service if leak confirmed.
  3. If memory remains >90% after step 2, trigger graceful restart: systemctl restart claw-gateway (drain connections first if load-balancer allows).
  4. Verify memory drops below 80% and CPU returns to <50% within 2 min post-restart.
  5. Trigger on-call database/backend team if memory pressure returns within 10 min (indicates upstream issue).

Watch

  • Memory usage on claw-gateway1 (target: <80%); alert if creeps back above 85% within 30 min.
  • Error rate and latency on claw-gateway endpoints; spike indicates incomplete recovery.

Escalate If

Memory does not drop below 85% within 15 minutes of restart, or error rate spikes >5% during recovery.

△
STATUS CHANGE
2026-07-26 16:50 UTC
OPEN -> RESOLVED (closed after validation; public manual form disabled and host pressure was transient)
Update Status
Context Sources
RAG Runbook unavailable ↗
Incident Logger fetched ↗
Infra Monitor fetched ↗
Details
ID #16
Severity HIGH
Source infra_monitor
Status RESOLVED
Opened 2026-07-26 16:34