patternbashkubernetesModerate
Node affinity and taints: controlling pod placement
Viewed 0 times
tainttolerationnode affinitypod schedulinggpu nodenode selectornoschedulenoexecuteprefernoschedulercluster topology
Error Messages
0/3 nodes are available: 3 node(s) had untolerated taintProblem
GPU workloads land on CPU-only nodes, or critical pods compete with batch jobs on the same nodes, wasting expensive resources or causing performance degradation.
Solution
Use taints + tolerations to repel pods from nodes, and node affinity to attract pods to specific nodes.
# Taint a node to repel general workloads
kubectl taint nodes gpu-node-1 gpu=true:NoSchedule
# Label a node for affinity selection
kubectl label nodes gpu-node-1 node-type=gpu# Pod spec: tolerate the taint AND prefer GPU nodes
spec:
tolerations:
- key: gpu
operator: Equal
value: "true"
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values: [gpu]Why
Taints mark a node as unsuitable for general pods unless the pod has a matching toleration. Node affinity provides fine-grained selection rules. Together they create a push-pull system: taints push away, affinity pulls toward.
Gotchas
- Taints have three effects: NoSchedule (don't place), PreferNoSchedule (try to avoid), NoExecute (evict existing pods too)
- NoExecute taints will evict already-running pods unless they have matching tolerations
- requiredDuringScheduling is a hard rule — pods stay pending if no node matches; preferredDuringScheduling is a soft preference
- DaemonSets automatically get tolerations for all NoSchedule and NoExecute taints unless you configure otherwise
- Node selectors (nodeSelector field) are simpler but less expressive than node affinity
Context
Workload placement optimization for heterogeneous node pools (GPU, high-memory, spot instances)
Revisions (0)
No revisions yet.