Troubleshooting GPU Scheduling Issues in Amazon EKS

A recent engagement with a client revealed a critical issue in their Amazon Elastic Kubernetes Service (EKS) cluster. Their GPU-enabled nodes (G4dn.xlarge instances) were failing to assign IP addresses to new pods, despite having available CPU and memory. The error message they encountered was:

"pod didn't trigger scale-up: 3 node(s) didn't match Pod's node affinity/selector, 6 max node group size reached."

This post walks through our troubleshooting approach, key discoveries, and the solution that resolved the problem.

Problem Analysis

Client’s Challenges

• The Amazon EKS cluster was hosting GPU workloads on G4dn.xlarge instances.
• Only one pod was running per GPU node, despite available resources.
• New pods failed to schedule due to node affinity and scaling constraints.

Initial Investigation Findings

• Node Resource Allocation: Each G4dn.xlarge instance provides 1 GPU, 4 vCPUs, and 16GB memory. Kubernetes was allocating an entire GPU per pod unless explicitly configured otherwise.
• Pod Affinity & Selector Rules: Anti-affinity and node selector configurations were correct and non-conflicting.

Potential Root Causes

• GPU Resource Requests: Pods requesting full GPU resources, restricted scheduling to one pod per node.
• ENI Limitations: Initially considered but ruled out after further analysis.

Root Cause: GPU Resource Allocation

The primary issue was that each pod was requesting a full GPU, preventing multiple pods from running on a single node. Since the client needed multiple pods to share a GPU, we implemented GPU time-slicing using the NVIDIA device plugin.

Understanding GPU Time-Slicing 

• Enables multiple pods to share a single GPU by time-division multiplexing.
• No memory isolation—a crashing pod may affect others sharing the GPU.
• Ideal for lightweight workloads that don’t require dedicated GPU resources.

Solution: Implementing GPU Time-Slicing

Step 1: Uninstall the Existing NVIDIA Device Plugin
Ensured the previous plugin was removed before applying the new configuration.

Step 2: Configure GPU Sharing via ConfigMap
Defined a ConfigMap to allow 4 pods per GPU:

Step 3: Deploy the Updated NVIDIA Device Plugin via Helm

Step 4: Apply Node Labels and Taints

To ensure only GPU workloads are scheduled on these nodes:

Label the Node:

(Required for newer NVIDIA plugin versions.)

Taint the Node (Recommended):

(Prevents non-GPU workloads from using GPU nodes.)

Best Practices & Recommendations

1. Use EKS-Optimized AMIs for GPU Workloads 

The client was using a custom AMI (Amazon Linux 2). We recommended:
Switching to the Amazon Linux 2023 NVIDIA AMI (pre-configured with NVIDIA drivers).

Reverse-engineering the AMI for custom builds, if needed.

2. Right-Sizing GPU Instances

For compute-heavy workloads, consider larger instances (e.g., G4dn.2xlarge).
For lightweight workloads, time-slicing helps reduce costs by maximizing GPU utilization.

Key Takeaways

By enabling GPU time-slicing, we allowed multiple pods to share a single GPU, resolving the scheduling bottleneck. Key insights:

• Default GPU requests assign 1 GPU per pod—adjust via time-slicing if sharing is needed.
• Always label GPU nodes (nvidia.com/gpu.present=true) for compatibility with newer NVIDIA plugins.
• Leverage EKS-optimized AMIs for reliable GPU support.

References

• Time-Slicing GPUs in Kubernetes
• NVIDIA Device Plugin for Kubernetes
• AWS EKS GPU Support