patternkubernetesMinor
Kubernetes pod eviction and traffic draining for to be evicted pod
Viewed 0 times
evictiontraffickubernetesfordrainingandpodevicted
Problem
I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.
It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.
During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.
I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.
I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
Also what are the best practices to fine tune Kubernetes pod eviction.
It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.
During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.
I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.
I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
Also what are the best practices to fine tune Kubernetes pod eviction.
Solution
setup a readiness Probe in the pod spec, something like this
readiness probes specifically inform service routing decisions
readinessProbe:
httpGet:
path: /
port: 443readiness probes specifically inform service routing decisions
Code Snippets
readinessProbe:
httpGet:
path: /
port: 443Context
StackExchange DevOps Q#14692, answer score: 1
Revisions (0)
No revisions yet.