ADR-035 A cluster moves from etcd with its revisions, while the API server is stopped

ADR-009 left open whether the store keeps etcd's revision numbers when a cluster moves onto it. It keeps them. quantykube migrate reads every key of the etcd at one revision, page by page at that revision, and writes them into a new store file with their own create and mod revisions and versions, the store's head at that revision and everything before it compacted. The leases the keys hold come along with the TTL they were granted, and get it in full, as after any restart of etcd; one that ended after the read gets a second, so its keys go as they would have in etcd.

Kept revisions are what make the move invisible to clients: every resourceVersion a controller or kubelet holds stays valid, and every write after the move gets a higher one. With numbers starting over, a client holding a resourceVersion from before would be ahead of the store and be refused until the store caught up with it.

Nothing writes while it copies: the API server is the only client of a kubeadm cluster's etcd, and it is stopped first by moving its static pod manifest out of the kubelet's directory. etcd itself stays as it was, untouched, so the way back is to point the API server at it again. migrate refuses a store file that exists, so a cluster is never copied over another.

In CI (scripts/migrate.sh) an unmodified kube-apiserver writes into a real etcd, is stopped, the keys are copied, and the API server started on QuantyKube finds every object at its resourceVersion, events with their lease among them.

The cost: the API server is down for the copy and its restart, seconds for a cluster of the usual size. Its clients' watches are told to list again, as after any restart of the API server, since its watch cache starts anew. History in QuantyKube begins at the move. The way back copies every key from the store into etcd, which numbers writes itself: it is first written to until its revision passes the store's, so no client is left holding a resourceVersion etcd has not reached. That costs one write per revision the store went past etcd, and the copy into etcd is not atomic; the API server stays stopped until it is done, and stays on the store if it fails.

How a node is moved: quantykube install store, as root on a kubeadm control plane node, does it from files alone. It keeps the API server's manifest in a journal under /var/lib/quantykube/install, moves the manifest out of the kubelet's directory and waits until no kube-apiserver process is left, migrates from the etcd the manifest names, with the API server's own etcd client certificate, starts the store as the systemd unit quantykube-store on 127.0.0.1:2479 with the certificates kubeadm made for etcd, and puts the manifest back with only --etcd-servers changed. It writes each step to the journal before it takes it, and on any failure after the API server was stopped it puts it back on etcd. quantykube undo store does that from the journal alone, without the API server, and keeps the store's file under a new name. The kubectl apply of ADR-009 is to run this command on the node; the command is what CI tests (scripts/kubeadm.sh move).