Kubernetes & EKS Pod Rightsizing Guide

By Illusio Platform Engineering Team · Last reviewed: 2026 · 10 min read

In most Kubernetes clusters, between 60% and 80% of requested CPU and memory is never actually utilized by workloads. Because the Kubernetes scheduler places pods based on requests rather than actual consumption, bloated manifests force cluster autoscalers to spin up dozens of unnecessary EC2 worker nodes.

The Request vs. Limit Distinction

Configuring container resources requires a clear understanding of the Linux cgroups mechanics behind them:

• Requests (Scheduling Guarantee): The minimum guaranteed resource allocation used by kube-scheduler to choose a worker node. If a pod requests 2 vCPUs and 4GB RAM, it will only be scheduled onto a node with at least that much allocatable capacity.
• Limits (Hard Ceiling): The maximum resource cap enforced at runtime. For CPU, exceeding limits triggers Linux Completely Fair Scheduler (CFS) quota throttling, causing severe latency spikes. For memory, exceeding limits triggers immediate kernel Out-Of-Memory termination (OOMKill, exit code 137).

Why Setting Hard CPU Limits Often Causes Outages

Many engineering teams mistakenly set limits.cpu equal to requests.cpu. When a microservice receives a burst of traffic or initiates garbage collection, the CFS period quota caps CPU usage mid-request, causing p99 latency to spike from 15ms to 1,200ms. In modern high-throughput architectures, best practice is to set conservative CPU requests with adequate buffer, and either omit CPU limits entirely or set them 3×–5× higher than requests to allow burstable execution without artificial throttling.

Methodology: Using VPA in Recommendation Mode

Rather than guessing resource values or relying on developer intuition, deploy the Vertical Pod Autoscaler (VPA) in recommendation-only mode (updateMode: "Off"). In this mode, VPA continuously analyzes Prometheus telemetry and outputs mathematically backed recommendations for target, lower-bound, and upper-bound requests without restarting running pods:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payment-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: payment-service
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4000m
          memory: 8Gi

Headroom Sizing for Production Reliability

A reliable rightsizing formula for steady-state production microservices:

• CPU Request: Set to P95 historical usage over a 14-day rolling window + 20% buffer for routine spikes. Pair with Horizontal Pod Autoscaler (HPA) targeting 65%–75% CPU utilization.
• Memory Request & Limit: Set request to P99 peak usage over 30 days + 25% safety buffer. Because memory cannot be throttled like CPU, set the memory limit 15%–20% above the request to provide an emergency safety envelope before triggering an OOMKill.

Automating Telemetry with Goldilocks

Tools like Fairwinds Goldilocks create dashboards on top of VPA recommendations across all namespaces. Engineering leads can quickly scan which teams have 5× overprovisioned manifests, set Namespace ResourceQuotas to prevent rogue configurations, and safely compress cluster footprint.