HiveBrain v1.2.0
Get Started
← Back to all entries
patternkubernetesMinor

Kubernetes pod eviction and traffic draining for to be evicted pod

Submitted by: @import:stackexchange-devops··
0
Viewed 0 times
evictiontraffickubernetesfordrainingandpodevicted

Problem

I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.

It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.

During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.

I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.

I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
Also what are the best practices to fine tune Kubernetes pod eviction.

Solution

setup a readiness Probe in the pod spec, something like this

readinessProbe:
  httpGet:
    path: /
    port: 443


readiness probes specifically inform service routing decisions

Code Snippets

readinessProbe:
  httpGet:
    path: /
    port: 443

Context

StackExchange DevOps Q#14692, answer score: 1

Revisions (0)

No revisions yet.