principlejavascriptTip
USE method for infrastructure resources: Utilization, Saturation, Errors
Viewed 0 times
USE methodBrendan Greggutilization saturation errorsCPU run queuenode_loadconnection poolbottleneck analysis
Problem
Performance issues in infrastructure resources (CPU, memory, disk, network, database connections) are diagnosed by guessing which resource is the bottleneck. Without a systematic approach, engineers spend hours checking the wrong resources.
Solution
Apply Brendan Gregg's USE method to every infrastructure resource:
For each resource in your system (CPU, memory, disks, network interfaces, DB connection pool), check all three. The first resource where any signal is elevated is your bottleneck.
- Utilization — the average time the resource is busy (e.g., CPU at 80%)
- Saturation — the degree to which the resource has extra work it cannot service (e.g., CPU run-queue length)
- Errors — count of error events (e.g., disk I/O errors)
For each resource in your system (CPU, memory, disks, network interfaces, DB connection pool), check all three. The first resource where any signal is elevated is your bottleneck.
# CPU utilization
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))
# CPU saturation (run queue pressure)
node_load1 > count(node_cpu_seconds_total{mode="idle"})
# Memory utilization
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
# Network errors
rate(node_network_receive_errs_total[5m]) > 0Why
USE gives a systematic checklist that prevents premature conclusions. It is resource-oriented rather than service-oriented, making it ideal for diagnosing underlying infrastructure problems that manifest as application symptoms.
Gotchas
- 100% CPU utilization is not always saturation — check run queue length (node_load) to confirm
- Memory utilization must use MemAvailable (not MemFree) which accounts for reclaimable cache
- For connection pools, saturation = requests waiting for a connection, errors = connection failures
- USE is complementary to RED — use RED at the service layer, USE at the infrastructure layer
Context
Diagnosing performance degradation caused by infrastructure resource exhaustion
Revisions (0)
No revisions yet.