QuantyKube

Move a cluster

All pages
On this page

One kubectl apply moves a running kubeadm cluster from etcd onto QuantyKube's store, and nodes you label from their runtime onto QuantyKube's. One patch moves everything back. etcd and containerd keep running, untouched, the whole time.

CI runs this on a one-node kubeadm cluster with the published image of every commit on main. It has not run in production anywhere. Read the status before you point it at a cluster you care about.

Before you apply

  • A cluster set up by kubeadm, with one control plane node. Other distributions and managed clusters are not supported yet (ADR-029, ADR-040).
  • Nodes for the runtime need cgroup v2 and the CNI plugins.
  • Read deploy/quantykube.yaml first. The pods it starts on nodes are root there, as any runtime installer is (ADR-039).

The move

Label each node that should run on QuantyKube's runtime, apply the manifest, and watch:

kubectl label node w1 install.quantyroot.io/runtime=quantykube
kubectl apply -f https://raw.githubusercontent.com/QuantyRoot/QuantyKube/<commit>/deploy/quantykube.yaml
kubectl get installation quantykube -w

Put a commit of main in place of <commit>, so that what you apply does not change under you. Nodes without the label stay on their runtime.

What happens

The manifest holds a namespace, the Installation custom resource, the operator's service account and RBAC, and the operator's Deployment (ADR-040). The operator creates the Installation named quantykube, asking for the store and for the runtime on every labelled node, and works in passes a few seconds apart, one step per pass:

  1. The store. Through a pod on the control plane node, quantykube install store stops the API server by moving its static pod manifest out of the kubelet's directory, copies every key of etcd at one revision into the store with etcd's own revisions, starts the store as the systemd unit quantykube-store on 127.0.0.1:2479 with the certificates kubeadm made for etcd, and puts the manifest back with only --etcd-servers changed (ADR-035).
  2. Each labelled node, in turn. Cordoned, drained as kubectl drain drains, then quantykube install runtime stops the kubelet and every pod of the old runtime, starts the systemd unit quantykube-runtime, points the kubelet at it and starts it again (ADR-038). The node is done once it is Ready on the new runtime, and is uncordoned.

Because revisions are kept, every resourceVersion a controller or kubelet holds stays valid across the move. Each step is written to the Installation's status before the next one starts, so an operator that is restarted halfway, as it is when its own node changes runtime, reads the status and goes on.

Watching it

kubectl get installation quantykube shows the phase and what the operator is doing. A node whose move neither finishes nor fails visibly is given ten minutes; then the installation is marked Failed and the node is left as its own undo left it.

Back to how it was

kubectl patch installation quantykube --type merge -p '{"spec":{"undo":true}}'

The undo runs the same steps in reverse, nodes first. etcd was never changed, so the API server goes back to it, with what was written on the store copied into etcd first. etcd numbers writes itself, so it is written to until its revision passes the store's: no client is left holding a resourceVersion etcd has not reached. The API server stays stopped until the copy is done, and stays on the store if the copy fails.

What it costs

  • The API server is down while the keys are copied and it restarts: seconds for a cluster of the usual size. Its clients' watches are told to list again, as after any restart of the API server.
  • History in QuantyKube begins at the move.
  • Every pod left on a moved node restarts, and the kubelet's cgroup driver changes from systemd to cgroupfs. A kubeadm upgrade rewrites the kubelet's configuration and needs the move done again.
  • The installer's pod is root on the node while a step runs. It does not stay behind as a DaemonSet.

Without the operator

The operator only runs commands that also work by hand, as root on the node: quantykube install store and quantykube undo store on the control plane node, quantykube install runtime and quantykube undo runtime on a drained node. Each keeps a journal of its steps under /var/lib/quantykube/install and puts everything back on a failure. CI tests them on their own (scripts/kubeadm.sh move and scripts/kubeadm.sh node).