QuantyKube

ADR-038 A node moves onto the runtime with its kubelet stopped and its old pods cleared

All pages

ADR-038, from docs/DECISIONS.md in the QuantyKube repository.

Status In force

quantykube install runtime, as root on a kubeadm node, moves its kubelet from containerd onto quantykube runtime, from files and systemd alone, as install store does for the API server (ADR-035). It keeps the kubelet's kubeadm-flags.env and config.yaml in a journal, stops the kubelet, stops and removes every pod of the old runtime over its CRI socket, starts the runtime as the systemd unit quantykube-runtime, points the kubelet at it with the cgroupfs driver the runtime places containers by, and starts the kubelet again. Any failure puts everything back; quantykube undo runtime does the same from the journal.

The old runtime's pods are cleared, not left running: the kubelet on the new runtime does not know them and would start every static pod a second time beside its first, on the same ports. Pods that are not DaemonSets or static pods are drained first by whoever runs the command; the command itself evicts nothing, because the node's own credentials may not evict other pods. kubeadm 1.27 and later keep the runtime's endpoint in config.yaml, earlier ones in the flags; both are read and changed.

The cost: every pod left on the node restarts, the control plane's among them on a control plane node. The kubelet's cgroup driver changes from systemd to cgroupfs, as k3s runs it; a kubeadm upgrade rewrites config.yaml and needs the move done again. CI moves a one-node kubeadm cluster, control plane, etcd, kube-proxy and CoreDNS, onto the runtime and back (scripts/kubeadm.sh node).