patternkubernetesMinor
Kubernetes readinessProbe: do health-checks run forever?
Viewed 0 times
checkskubernetesreadinessprobehealthforeverrun
Problem
I have some doubts about the HTTP readinessProbe:
- Suppose that the probe fails several times (failureThreshold) and a pod is marked as NOT READY. Will k8s keep trying the readinessProbe forever and eventually mark the pod as Ready if it begins to respond correctly? Or a pod that is marked as Not Ready is left there forever without retrying?
- In case the Not Ready pod is left there forever, will the K8s Deployment try to start new pods? Or I will be left with less pods available?
- If a pod passes the readinessProbe, will k8s keep checking if it is ready and eventually (if it begins to fail) mark it as not ready in the future? Is that possible? Or a pod that is marked once as ready is ready forever?
Solution
1 - yes, containers are left there forever. Think of readiness probe as a breaker, if the app gets overwhelmed it cuts the traffic to the pod until it's ready to get traffic again.
2 - no, by itself, readiness probes only stop/start traffic to a pod. To signal kubernetes that the pod needs to be restarted use liveness probes.
3 - yes, as per answer 1. Readiness probes are a way of letting kubernetes know that the pod can handle traffic or if they need a break.
probe: startup probes - waits for the app to start
use case: apps that need some time before start serving traffic, for example, building assets on start, pre-fetching assets, building caches, etc.
liveness probes - checks if the app is healthy
use case: check if the app is running, if the app has exited due to an exception and needs to be restarted, or if the app is running but can't serve requests
readiness probes - checks if the app can handle traffic
use case: apps that can arbitrarily stop receiving traffic, e.g. maintenance mode, data intensive apps that can degrade in performance and need a way to
You should use liveness and readiness probes always, use startup probes when your app takes more time than usual to start.
Example:
the /health endpoint queries the database using a simple query (select id, email from user limit 1) and returns a json
if the pod loses connectivity to the database, the liveness probe will fail and the pod will be restarted.
if the database is being overwhelmed by requests, the query will take more than 1s, the probe will mark the pod as not ready, preventing that additional requests reach the database and preventing it from progressing from overwhelmed to crashed. once the database is able to catch up and starts returning queries in less than 1s, readiness probe will mark the pod as healthy again and kubernetes will resume serving requests to the pod.
2 - no, by itself, readiness probes only stop/start traffic to a pod. To signal kubernetes that the pod needs to be restarted use liveness probes.
3 - yes, as per answer 1. Readiness probes are a way of letting kubernetes know that the pod can handle traffic or if they need a break.
probe: startup probes - waits for the app to start
use case: apps that need some time before start serving traffic, for example, building assets on start, pre-fetching assets, building caches, etc.
liveness probes - checks if the app is healthy
use case: check if the app is running, if the app has exited due to an exception and needs to be restarted, or if the app is running but can't serve requests
readiness probes - checks if the app can handle traffic
use case: apps that can arbitrarily stop receiving traffic, e.g. maintenance mode, data intensive apps that can degrade in performance and need a way to
You should use liveness and readiness probes always, use startup probes when your app takes more time than usual to start.
Example:
- python app with postgres database:
- liveness probe: get /health status 200
- readiness probe: get /health status 200, timeout 1s
the /health endpoint queries the database using a simple query (select id, email from user limit 1) and returns a json
if the pod loses connectivity to the database, the liveness probe will fail and the pod will be restarted.
if the database is being overwhelmed by requests, the query will take more than 1s, the probe will mark the pod as not ready, preventing that additional requests reach the database and preventing it from progressing from overwhelmed to crashed. once the database is able to catch up and starts returning queries in less than 1s, readiness probe will mark the pod as healthy again and kubernetes will resume serving requests to the pod.
Context
StackExchange DevOps Q#14950, answer score: 2
Revisions (0)
No revisions yet.