snippetkubernetesMinor
How to introduce(join) new worker node to an existing k8s cluster with selective scheduling only
Viewed 0 times
introduceschedulingnewselectivewithnodek8sworkerjoinhow
Problem
Given a working Kubernetes cluster consisting of a master and some workers.
We need to add a node to run a very specific pod and be part of the cluster for networking reasons. Then being largely ignored for later pod creation by the master.
Adding selectors to every deployment to avoid this node is out of question.
Thanks!
We need to add a node to run a very specific pod and be part of the cluster for networking reasons. Then being largely ignored for later pod creation by the master.
Adding selectors to every deployment to avoid this node is out of question.
Thanks!
Solution
Possibly you are looking for this feature which is closed with no action since there are ways to work around this problem.
kubeadm already allows to set taints for the joining node using the configuration file at join time (check my other answer for details on how to do that)
One Other way to achieve this is by running join as usual but taking few extra steps as below after join
1) After successful join , Force Drain the new node to make sure it removes unwanted pod which might have scaled to it post join operation
2) Drain command will auto cordon off the node. (Note : also consider special handling to remove any daemonset which might have scaled to this node after join)
3) Now Taint the new node as needed and then uncordon it
4) Add nodeselector plus tolerance to pods you want to be scheduled on this new node.
Logs of above steps for reference are listed below.
On below example i have joined a new node-02 and assume that some pod were scaled in an existing deployment so new node got used immediately on join, so before adding we need drain action and then taint the node so going forward no scheduling can be done unless pod has tolerance to the taint added
Nodes Before Drain or Taint
Drain The node
Taint the new node
Now we are good to uncordon the node, no more pods should be able to use this noe unless they add toleration to the new taint.
We test by scaling the deployment and nothign should be added on node02
kubeadm already allows to set taints for the joining node using the configuration file at join time (check my other answer for details on how to do that)
One Other way to achieve this is by running join as usual but taking few extra steps as below after join
1) After successful join , Force Drain the new node to make sure it removes unwanted pod which might have scaled to it post join operation
kubectl drain --force 2) Drain command will auto cordon off the node. (Note : also consider special handling to remove any daemonset which might have scaled to this node after join)
3) Now Taint the new node as needed and then uncordon it
4) Add nodeselector plus tolerance to pods you want to be scheduled on this new node.
Logs of above steps for reference are listed below.
On below example i have joined a new node-02 and assume that some pod were scaled in an existing deployment so new node got used immediately on join, so before adding we need drain action and then taint the node so going forward no scheduling can be done unless pod has tolerance to the taint added
Nodes Before Drain or Taint
$ kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-86c57db685-8cdkx 1/1 Running 0 27s 192.168.58.120 k8s-node02-calico
pod/nginx-86c57db685-xnh6q 1/1 Running 0 65s 192.168.182.78 k8s-node01-calico
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 443/TCP 2d
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx 2/2 2 2 65s nginx nginx app=nginx
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-86c57db685 2 2 2 65s nginx nginx app=nginx,pod-template-hash=86c57db685Drain The node
$ kubectl drain k8s-node02-calico
node/k8s-node02-calico cordoned
evicting pod "nginx-86c57db685-8cdkx"
pod/nginx-86c57db685-8cdkx evicted
node/k8s-node02-calico evicted
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-calico Ready master 2d v1.17.2
k8s-node01-calico Ready 2d v1.17.2
k8s-node02-calico Ready,SchedulingDisabled 2d v1.17.2Taint the new node
$ kubectl taint node k8s-node02-calico newnode=nobodyallowed:NoSchedule
node/k8s-node02-calico taintedNow we are good to uncordon the node, no more pods should be able to use this noe unless they add toleration to the new taint.
$ kubectl uncordon k8s-node02-calico
node/k8s-node02-calico uncordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-calico Ready master 2d v1.17.2
k8s-node01-calico Ready 2d v1.17.2
k8s-node02-calico Ready 2d v1.17.2We test by scaling the deployment and nothign should be added on node02
$ kubectl scale deployment --replicas=3 nginx
deployment.apps/nginx scaled
$ kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-86c57db685-vm8ql 1/1 Running 0 3m30s 192.168.182.75 k8s-node01-calico
pod/nginx-86c57db685-xnh6q 1/1 Running 0 10m 192.168.182.78 k8s-node01-calico
pod/nginx-86c57db685-zh2sj 1/1 Running 0 8m19s 192.168.182.76 k8s-node01-calico
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 443/TCP 2d
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx 3/3 3 3 10m nginx nginx app=nginx
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-86c57db685 3 3 3 10m nginx nginx app=nginx,pod-template-hash=86c57db685Code Snippets
$ kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-86c57db685-8cdkx 1/1 Running 0 27s 192.168.58.120 k8s-node02-calico <none> <none>
pod/nginx-86c57db685-xnh6q 1/1 Running 0 65s 192.168.182.78 k8s-node01-calico <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d <none>
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx 2/2 2 2 65s nginx nginx app=nginx
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-86c57db685 2 2 2 65s nginx nginx app=nginx,pod-template-hash=86c57db685$ kubectl drain k8s-node02-calico
node/k8s-node02-calico cordoned
evicting pod "nginx-86c57db685-8cdkx"
pod/nginx-86c57db685-8cdkx evicted
node/k8s-node02-calico evicted
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-calico Ready master 2d v1.17.2
k8s-node01-calico Ready <none> 2d v1.17.2
k8s-node02-calico Ready,SchedulingDisabled <none> 2d v1.17.2$ kubectl taint node k8s-node02-calico newnode=nobodyallowed:NoSchedule
node/k8s-node02-calico tainted$ kubectl uncordon k8s-node02-calico
node/k8s-node02-calico uncordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-calico Ready master 2d v1.17.2
k8s-node01-calico Ready <none> 2d v1.17.2
k8s-node02-calico Ready <none> 2d v1.17.2$ kubectl scale deployment --replicas=3 nginx
deployment.apps/nginx scaled
$ kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-86c57db685-vm8ql 1/1 Running 0 3m30s 192.168.182.75 k8s-node01-calico <none> <none>
pod/nginx-86c57db685-xnh6q 1/1 Running 0 10m 192.168.182.78 k8s-node01-calico <none> <none>
pod/nginx-86c57db685-zh2sj 1/1 Running 0 8m19s 192.168.182.76 k8s-node01-calico <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d <none>
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx 3/3 3 3 10m nginx nginx app=nginx
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-86c57db685 3 3 3 10m nginx nginx app=nginx,pod-template-hash=86c57db685Context
StackExchange DevOps Q#10707, answer score: 2
Revisions (0)
No revisions yet.