AWS Cost Anomaly Detection & Alerting Playbook

By Illusio Platform Engineering Team · Last reviewed: 2026 · 8 min read

A recursive Lambda loop, a runaway data migration, or an unthrottled Kubernetes ingress can run up tens of thousands of dollars in unintended cloud spend within a single weekend. Static monthly budgets catch these disasters weeks too late.

Why Static AWS Budgets Fail

Traditional AWS Budgets check total accumulated spend against a fixed monthly threshold (e.g. $50,000). If a rogue script starts burning $1,500/hour on day 3, the monthly budget alert will not fire until day 20, when the account has already consumed $35,000 in excess charges.

AWS Cost Anomaly Detection replaces static thresholds with machine learning models trained on your account’s unique hourly and daily spending rhythms. It distinguishes between normal organic growth and abnormal rate-of-change spikes, notifying on-call engineers within 2 to 6 hours of an anomaly.

Setting Up Monitors in Terraform

Deploying anomaly detection natively through Infrastructure as Code ensures coverage across all active AWS accounts:

resource "aws_ce_anomaly_monitor" "service_monitor" {
  name              = "AWS-Services-Anomaly-Monitor"
  monitor_type      = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "critical_alerts" {
  name      = "Realtime-Cost-Alerts-Slack"
  frequency = "IMMEDIATE"

  monitor_arn_list = [
    aws_ce_anomaly_monitor.service_monitor.arn
  ]

  subscriber {
    type    = "SNS"
    address = aws_sns_topic.cost_anomalies.arn
  }

  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_PERCENTAGE"
      values        = ["30"]
      match_options = ["GREATER_THAN_OR_EQUAL"]
    }
  }
}

Routing Alerts to Engineering Channels

Sending cost alerts to an unmonitored accounting email inbox guarantees delay. For actionable containment:

• AWS Chatbot / Slack Integration: Connect the SNS topic to an active engineering channel (e.g. #ops-cloud-alerts). The alert includes the root cause service, estimated dollar impact, and direct AWS console deep links.
• PagerDuty Escalation for High-Impact Anomalies: Configure a secondary subscription with a high dollar impact threshold (e.g. >$500 single-event impact) to page the on-call platform engineer immediately.

Rapid Incident Triage Checklist

When an anomaly alert fires, on-call engineers should execute this standard response protocol:

1. Identify Service & Region: Check the alert payload for the offending AWS service (e.g. CloudWatch Logs, NAT Gateway, DynamoDB, EC2).
2. Correlate Recent Deployments: Inspect CI/CD deploy logs over the last 4 hours to verify if a recent release altered log levels (e.g. debug logging to CloudWatch) or infinite retry loops.
3. Isolate Rogue Capacity: If an autoscaling group or Lambda function is spinning out of control, apply immediate maximum scaling limits or throttle concurrency.
4. Engage AWS Support: If the anomaly resulted from an external DDoS amplification or an AWS service failure, open a high-severity billing support ticket immediately to request service credit remediation.