Kubernetes system upgrade

 Node upgrade

Node down > 5m: 

pod will be terminated  in it

Replicaset will be automatically recreate on other nodes

Single node will be terminated

If node down < 5m: directly do it

If node down > 5m: 

Before #kubectl drain <node name> --ignore-daemonsets

After #kubectl uncordon <node name>

Description:

#kubectl drain <node name>: recreated pods in other nods (only replicaset / no single pod) and mask node as unavailable for new pods

#kubectl cordon <node name>: mask node as unavailable for new pods

#kubectl uncordon <node name>: mask node as available for new pods

Best solution:

1. Check pod type: single pod / replicaset

2. If single pod try to create in other nodes (cordon the node plan for maintenance)

3. Drain the node for maintenance

4. After change finish

5. Uncordon the node for maintenance

K8S upgrade

Release: https://github.com/kubernetes/kubernetes/releases

Cluster upgrade

Master

1. kube-apiserver should have highest version 1.10

2. Controller-manager can be one version lower 1.9 / 1.10

3. Kube-scheduler can be one version lower 1.9 / 1.10

4. Kubelet can be two version lower 1.8/1.9/1.10

5. Kube-proxy can be two version lower 1.8/1.9/1.10

Work Node: kubectl can be one version higher: 1.9 / 1.10 / 1.11

K8S only support three version 

For example v1.12(latest) / v1.11 / v1.10

Upgrade one release by one release 1.10 --> 1.11 --> 1.12

Cluster upgrade: upgrade master node; upgrade work node one by one

Google cloud --> click "Upgrade" button

Kubeadm setup cluster 

1. Master node from v1.11.0

Check the upgrade target version #kubeadm upgrade plan

Drain the master node #kubectl drain master --ignore-daemonsets

Upgrade kubeadm #apt install kubeadm=1.12.0-00

Upgrade master node #kubeadm upgrade apply v1.12.0

Upgrade kubelet #apt install kubelet=1.12.0-00

Release the master node #kubectl uncordon master

2. Work node (one by one)

Drain work node #kubectl drain <work node> --ignore-daemonsets

Ssh to work node #ssh <work node>

Upgrade kubeadm #apt install kubeadm=1.12.0-00

Upgrade kubelet #apt install kubelet=1.12.0-00

Upgrade work node #kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)

Backup & Restore

Declarative method: 

Make yaml file --> save in github --> create item in yaml file

Backup item #kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml

Etcd backup #ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \

     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \

     snapshot save /tmp/snapshot-pre-boot.db