Prometheus metrics instrumentation patterns

Need to instrument application code for monitoring with Prometheus.

Four types of Prometheus metrics:

from prometheus_client import Counter, Histogram, Gauge, Summary

# Counter: things that only go up
http_requests = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
http_requests.labels('GET', '/api/users', '200').inc()

# Histogram: distribution of values (latency, sizes)
request_duration = Histogram(
    'http_request_duration_seconds',
    'Request duration in seconds',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

# Use as decorator or context manager
@request_duration.labels('/api/users').time()
def handle_users():
    ...

# Gauge: values that go up and down
in_progress = Gauge(
    'requests_in_progress',
    'Number of requests in progress'
)
in_progress.inc()  # +1
in_progress.dec()  # -1

# RED method for services:
# Rate: http_requests_total
# Errors: http_requests_total{status=~"5.."}
# Duration: http_request_duration_seconds

# USE method for resources:
# Utilization: resource_usage_percent
# Saturation: queue_length
# Errors: resource_errors_total

Naming conventions:

• _total suffix for counters
• _seconds or _bytes for units
• Snake_case, lowercase
• Prefix with subsystem: http_, db_, cache_

Good metrics enable alerting, dashboarding, and root cause analysis. The RED and USE methods provide a systematic approach to what to measure.

Production services needing monitoring and alerting