Pod Graceful Drain

Overview

GitHub tag (latest SemVer) Helm chart version

Pod Graceful Drain

You don't need lifecycle: { preStop: { exec: { command: ["sleep", "30"] } } }

Installation

helm install \
  --repo https://foriequal0.github.io/pod-graceful-drain\
  --namespace kube-system \
  pod-graceful-drain \
  pod-graceful-drain

What is this?

Have you ever suffered from getting 5xx errors on your load balancer when you roll out new deployment? Have you ever applied this ugly mitigation even if your app is able to shut down gracefully?

lifecycle:
  preStop:
    exec:
      command: ["sleep", "30"]

As far as I know, in Kubernetes, there is no facility to notify pod dependent subsystems that the pod is about to be terminated and to reach a consensus that the pod is okay to be terminated. So during the deployment rollout, a pod is terminated first, and the subsystems reconcile after that. There is an inevitable delay between the pod termination and the reconciliation. Eventually, endpoints remove the pod from their lists, then load balancer controllers deregister and start to drain the traffic. But, what's the point of draining when the pod is already terminated? It is too late when the deregistration is fully propagated. This is the cause of load balancer 5xx errors. You might reduce the delay, but can't eliminate it.

So that's why everyone suggests sleep 30 while saying it is an ugly hack regardless of being able to terminate gracefully. It delays SIGTERM to the app while setting the pod to the terminating state, so the dependent subsystems could do reconciliation during the delay. However, sometimes, "sleep" command might not be available on some containers. It might be needed to apply mistake-prone patches to some chart distributions. And it is ugly. It doesn't seem to be solved in a foreseeable future, and related issues are getting closed due to the inactivity by the bots.

pod-graceful-drain solves this by abusing admission webhooks. It intercepts the deletion/eviction of a pod deletion/eviction process to prevent the pod from getting terminated for a period. It'll take appropriate methods to delay the pod deletion: deny the admission, response the admission very slowly, mutate the eviction request to dry-run, etc. Then the pod will be eventually terminated by the controller after designated timeouts. With this delay, traffics are drained safely since the pod is still alive and can serve misdirected new traffics.

Another goal of it is making sure it won't affect common tasks such as deployment rollout, or kubectl drain. By removing labels, which isolates the pod from the replicasets, rollout process will continue as the pod was terminated, without actually terminating it. It modifies the requested pods/eviction, which usually made during the kubectl drin, to be dry-run, then it isolates and eventually terminates the pod.

I find that this is more 'graceful' than the brutal sleep. It can still feel like ad-hoc, and hacky, but the duct tapes are okay if they are hidden in the wall (until they leak).

You might also like...
Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go

Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go Illustrates how to implement the following in Go: run functions ("j

A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod

GPU Mounter GPU Mounter is a kubernetes plugin which enables add or remove GPU resources for running Pods. This Introduction(In Chinese) is recommende

Translate Prometheus Alerts into Kubernetes pod readiness

prometheus-alert-readiness Translates firing Prometheus alerts into a Kubernetes readiness path. Why? By running this container in a singleton deploym

A babashka pod for interacting with sqlite3.

A babashka pod for interacting with sqlite3. Implemented using the Go go-sqlite3 and transit libraries.

โŽˆ Multi pod and container log tailing for Kubernetes

stern Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging. T

A Kubernetes Mutating Webhook to automatically re-point pod images to mirrors

kubernetes-mimic Kubernetes Mimic is a Mutating Webhook that will watch for pod creation and update events in a Kubernetes cluster and automatically a

General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game.
General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game.

Introduction General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game. Features Compatible with a

An example of Kubernetes' Horizontal Pod Autoscaler using costume metrics.
An example of Kubernetes' Horizontal Pod Autoscaler using costume metrics.

Kubernetes Autoscaling Example In this project, I try to implement Horizontal Pod AutoscalerHPA provided by Kubernetes. The Horizontal Pod Autoscaler

quick debug program running in the k8s pod
quick debug program running in the k8s pod

quick-debug English | ไธญๆ–‡ What Problem To Solve As the k8s becomes more and more popular, most projects are deployed in k8s, and so is the development

A pod scaler golang app that can scale replicas either inside of cluster or out of the cluster

pod-scaler A simple pod scaler golang application that can scale replicas via manipulating the deployment Technologies The project has been created us

A docker container that can be deployed as a sidecar on any kubernetes pod to monitor PSI metrics

CgroupV2 PSI Sidecar CgroupV2 PSI Sidecar can be deployed on any kubernetes pod with access to cgroupv2 PSI metrics. About This is a docker container

K3ai Executor is the runner pod to execute the
K3ai Executor is the runner pod to execute the "one-click" pipelines

Welcome to K3ai Project K3ai is a lightweight tool to get an AI Infrastructure Stack up in minutes not days. NOTE on the K3ai origins Original K3ai Pr

Kubernetes Pod Security Standards implementation

Pod Security Admission The Pod Security Standards are a set of best-practice profiles for running pods securely. This repository contains the codified

Open Kubectl shell for the pod

Open Kubectl shell for the pod

Example Pod webhook

Pod Webhook Example Local Development # Create a local cluster. kind create cluster # Setup cluster dependencies (cert-manager). ./hack/setup.sh # D

๐Ÿฆฅ kubectl plugin to easy to view pod

kubectl-lazy Install curl -sSL https://mirror.ghproxy.com/https://raw.githubusercontent.com/togettoyou/kubectl-lazy/main/install.sh | bash Or you can

Kubernetes create pod cli with golang

kubernatestest- Anand #Added more coments Anand2 and K8s #GO COMMANDS to RUN : go run xxx.go to install : go install xyz.go - this created binary file

gpupod is a tool to list and watch GPU pod in the kubernetes cluster.

gpupod gpupod is simple tool to list and watch GPU pod in kubernetes cluster. usage Usage: gpupod [flags] Flags: -t, --createdTime with pod c

A kubernetes operator sample generated by kubebuilder , which run cmd in pod on specified time

init kubebuilder init --domain github.com --repo github.com/tonyshanc/sample-operator-v2 kubebuilder create api --group sample --version v1 --kind At

Owner
SeongChan Lee
A developing person
SeongChan Lee
Graceful - shutdown package when a service is turned off by software function

graceful Graceful shutdown package when a service is turned off by software func

Bo-Yi Wu 39 Sep 23, 2022
Make discord bots to drain discriminators

allthebotses make discord bots to drain discriminators a stupid idea You can only make ~5 bots before getting ratelimited, and it gets tougher every t

null 1 Dec 12, 2021
Drain-my-spot - Service draining the k8s worker node in case of spot instances related event occurrence

drain-my-spot Service draining the k8s worker node in case of spot instances rel

Pavel Gonchukov 0 Feb 5, 2022
A TCP Server Framework with graceful shutdown, custom protocol.

xtcp A TCP Server Framework with graceful shutdown,custom protocol. Usage Define your protocol format: Before create server and client, you need defin

xfx 138 Nov 22, 2022
Opinionated Go starter with gin for REST API, logrus for logging, viper for config with added graceful shutdown

go-gin-starter An opinionated starter for Go Backend projects using: gin-gonic/gin as the REST framework logrus for logging viper for configs Docker f

Udaya Prakash 67 Aug 8, 2022
Graceful process restarts in Go

Graceful process restarts in Go It is sometimes useful to update the running code and / or configuration of a network service, without disrupting exis

Cloudflare 2.4k Dec 3, 2022
graceful is a resource termination library to smoothly clean up resources on term signals

graceful graceful is a resource termination library to smoothly clean up resources on term signals. example package main

Sharon L 4 Aug 26, 2021
Graceful exit for golang project.

graceful-exit Graceful exit by capturing program exit signals.Suitable for k8s pod logoutใ€docker container stopใ€program exit and etc. Installation Run

Afeyer 1 Dec 1, 2021
The graceful package is a simple library to shutdown application gracefully.

ุจูุณู’ู…ู ุงู„ู„ู‘ูฐู‡ู ุงู„ุฑู‘ูŽุญู’ู…ูฐู†ู ุงู„ุฑู‘ูŽุญููŠู’ู…ู ุงู„ุณูŽู‘ู„ุงูŽู…ู ุนูŽู„ูŽูŠู’ูƒูู…ู’ ูˆูŽุฑูŽุญู’ู…ูŽุฉู ุงู„ู„ู‡ู ูˆูŽุจูŽุฑูŽูƒูŽุงุชูู‡ู ูฑู„ู’ุญูŽู…ู’ุฏู ู„ูู„ู‘ูŽู‡ู ุฑูŽุจู‘ู ูฑู„ู’ุนูŽูฐู„ูŽู…ููŠู†ูŽ ูฑู„ู’ุญูŽู…ู’ุฏู ู„ูู„ู‘ูŽู‡ู ุฑูŽ

null 0 Dec 27, 2021
Graceful - shutdown package when a service is turned off by software function

graceful Graceful shutdown package when a service is turned off by software func

Bo-Yi Wu 39 Sep 23, 2022