Why Your Amazon EKS Pods Are Stuck in Pending Despite "Available" IPs

When running scaled Kubernetes workloads on Amazon EKS, you inevitably hit an IP address exhaustion issue. To keep clusters growing, AWS introduced a vital feature in the Amazon VPC CNI plugin: prefix delegation.

Instead of a node's Elastic Network Interface (ENI) requesting a single IP for every pod, the ENI is assigned an entire block of addresses, a prefix, typically a /28. This significantly accelerates pod IP assignment and boosts scalability.

But what happens when the system fails? We recently encountered a puzzling customer issue where pods could not obtain or maintain client IP addresses, despite the AWS console reporting a high number of available addresses. The catch is that the console shows the number of free individual IPs, not whether those addresses can form usable /28 prefix blocks. Scattered leftover IPs are useless if they cannot form a contiguous block. The pods were stuck in Pending, a classic symptom of IP shortage, yet the numbers contradicted the error.

This post details the debugging journey, from understanding the relationship between /25 and /28 subnets to uncovering the hidden root cause: subnet fragmentation.

The Customer Conundrum: IPs Not Assigning

Here was the setup and the confusing symptoms we faced:

• The Amazon EKS cluster was running in a /25 subnet.
• Prefix delegation mode was enabled in the CNI.
• The AWS console reported many free, available IPs in the subnet.
• Despite this, pods were stuck in Pending with the error: "failed to assign IP address."

The immediate conclusion was an IP shortage, but the official metrics suggested otherwise. This meant the visible IP count was a deceptive metric, requiring a deeper look into how prefixes are allocated.

Subnet Fundamentals: Why Block Size Matters

To understand the problem, you must think in terms of address blocks, not individual IPs.

The /25 and /28 Interaction

The CIDR (Classless Inter-Domain Routing) notation defines the size of an IP address range.

A /25 subnet (128 total IPs) can be perfectly divided into eight distinct /28 blocks (16 IPs each).

The critical realization here is that prefix delegation does not assign single IPs; it allocates entire, contiguous /28 chunks to the ENIs.

This was the key insight: The customer's subnet had many free, scattered individual IPs, but these leftover addresses were useless if a clean, contiguous /28 block was not available. The remaining free IPs were too fragmented to form a full block.

Our Debugging Walkthrough

Solving this required a methodical approach to eliminate common Amazon EKS issues, such as traffic spikes, and isolate the root cause.

• Initial Checks – Nodes were Ready, ruling out basic kubelet or network configuration failures.
• Subnet IP Deception – The console's "available IP" count was misleading. We suspected this metric did not reflect the reality of prefix availability.
• CNI Logs – We confirmed the aws-node DaemonSet logs showed prefix delegation was enabled, ruling out simple configuration errors.
• CloudTrail Investigation – We searched CloudTrail for prefix assignment API call failures. Finding none ruled out common IAM or permissions issues that often block ENI operations.

The Key Hypothesis: Fragmentation

After ruling out the standard culprits, we hypothesized: The issue isn't the total count of free IPs; it's the availability of contiguous /28 blocks. The subnet was suffering from fragmentation.

The Script That Revealed the Truth

To prove the hypothesis, we needed to see every IP currently attached to an ENI within the subnet. We used a simple AWS CLI script:

Running this script confirmed the hypothesis: The allocated IPs were scattered across the subnet's range, consuming portions of all eight potential /28 blocks. Subnet fragmentation was confirmed. Zero full, intact /28 blocks remained for a new ENI to be allocated.

Conclusion and Operator Takeaways

The root cause was definitive: Subnet fragmentation had made all remaining IPs unusable for prefix delegation.

The Fix

We worked with the customer on two main solutions:

• Free Up Usage – Releasing full ENIs or workloads to clear an entire /28 block for reuse
• Plan for Scale – Designing future subnets to be larger than /25 (e.g., /24 or /23) to provide more resilience against fragmentation

Once a clean /28 block was made available, the prefix delegation mechanism immediately worked, and pods began acquiring IPs without further issues.

Key Takeaways for Amazon EKS Operators

• Prefix Delegation Works in Blocks – When prefix delegation is enabled, you must think in terms of /28 blocks. Scattered free IPs are useless. You need a full, contiguous /28 block for allocation to the node.
• Subnet Design Matters – A /25 only provides eight /28 chunks. Larger subnets offer greater stability and flexibility as your cluster scales.
• The Console Can Mislead – The "available IP" count is a general metric. To truly diagnose prefix delegation issues, inspect ENI assignments directly to check for fragmentation.

If you use prefix delegation in Amazon EKS, your real bottleneck is not "free IP count." It is "available block count."