The Silent Bottleneck: Avoiding Subnet IP Exhaustion in Amazon EKS

If you’ve ever scaled an Amazon EKS cluster and suddenly found pods stuck in “Pending” with cryptic error messages, there’s a good chance you’ve run into subnet IP exhaustion. It’s one of those issues that doesn’t show up in small dev clusters, but once your workloads grow, it sneaks up fast and brings everything to a halt.

Let’s break down what’s really happening, why it matters, and how you can keep your clusters from running out of IPs.

What subnet IP exhaustion looks like

Here’s the story. You’re running an Amazon EKS cluster, things are humming along, then a deploy happens, and pods just won’t start.

You check the events:

• Insufficient pods
• Failed to create pod sandbox: failed to set up sandbox container network

At first glance, it looks like a Kubernetes scheduling issue. In reality, it’s not Kubernetes failing. It’s the underlying VPC subnets that have run out of usable IPs.

Why Amazon EKS eats through IPs so quickly

By default, AWS EKS uses the AWS VPC CNI plugin. That means:

• Each node gets one or more Elastic Network Interfaces (ENIs).
• Each ENI maps to a subnet and gets a pool of IPs.
• Every pod on that node gets a real VPC IP from that pool.

This is great for native VPC networking; pods can talk to anything in the VPC without NAT or overlays. The downside? You’re burning through subnet IPs at the same pace you’re spinning up pods.

Here’s the kicker:

• A /24 subnet gives you 251 usable IPs.
• That sounds like a lot until you realize every node reserves some for itself, and each pod also needs one.
• Add in scaling events, DaemonSets, and sidecars, and suddenly that /24 is bone dry.

The quick fixes

If you’re already stuck, here are the emergency levers:

Expand your subnets

a) Add larger subnets (/19 or /20 instead of /24).

b) Or attach secondary CIDR blocks to the VPC and create new subnets.

c) This buys you breathing room, but it’s really just a band-aid.

Spread pods across multiple subnets

a) Make sure your node groups are using more than one subnet per AZ.

b) This balances IP usage and avoids concentrating pressure on a single small subnet.

Smarter long-term strategies

If you don’t want to keep playing whack-a-mole with subnet sizes, here are better approaches:

• Prefix delegation (VPC CNI feature)
  Instead of assigning individual IPs, AWS can hand out /28 prefixes to nodes. Each prefix provides 16 pod IPs, which dramatically reduces ENI churn and subnet pressure.
• Custom networking mode
  Use secondary ENIs from a different subnet or CIDR just for pod traffic. This way your node management IPs and pod IPs don’t fight over the same pool.
• Consider IPv6
  If you’re planning for very large clusters, enabling IPv6 gives you essentially unlimited pod IP space. It’s still a bit of a shift operationally, but it’s future-proof.
• Overlay networking (e.g., Cilium)
  If you don’t need every pod to have a native VPC IP, an overlay CNI can decouple pod scaling from VPC IP limits.

How to know before it bites you

Subnet exhaustion is sneaky, but it leaves clues:

• Watch the aws-node DaemonSet logs for IP allocation errors.
• Set Amazon CloudWatch alarms on ENI IP usage.
• Track pod density per node and compare against subnet free IPs.

Proactive monitoring beats getting paged when half your pods are stuck in Pending.

Wrapping it up

Subnet IP exhaustion in Amazon EKS isn’t a bug; it’s just how the VPC CNI works. The problem is that most of us don’t think about subnet sizing until it breaks production.

The good news: once you understand how pods consume IPs, you can design around it. Start with larger CIDRs, enable prefix delegation, and keep monitoring subnet usage. That way, your scaling story is about smooth growth,  not a surprise bottleneck.