ADR-034 The store on Raft: every write a command in the log, the log in a file of its own

quantykube serve --node <id> --peers ... makes the store one node of a cluster. Every request that changes the store, Put, DeleteRange, Txn, Compact, LeaseGrant, LeaseRevoke and History's Restore, becomes a command in the Raft log: the method, the request as the client sent it, the time it was taken, and which node and caller wait for it. Every node applies the commands in log order through the same code that serves them on a single node, so revisions, answers and watch events are the same everywhere, and the node a client asked answers it when it applies the command itself. A follower passes commands to its leader inside Raft. Reads wait for a ReadIndex unless the client asked for a serializable read; watches are served from each node's own store.

Every write a command makes carries the command's Raft index into the store, in the same commit, so that a node restarting knows exactly what it already applied and applies nothing twice. A command that writes nothing leaves no index behind; applied again after a crash it meets the same state it met the first time, since nothing changed in between, and does nothing again. The check in CI: the killed leader comes back with the same revisions as the others. Without the index it came back three revisions ahead.

Leases follow etcd: the node that takes a grant picks the id, so every node grants the same lease; only the leader decides that a lease ran out, and revokes it through the log; a new leader first gives every lease its full TTL again. A keepalive refreshes the lease only on the node it reaches, which is the leader's for kube-apiserver in practice and not always for other clients.

The Raft log lives in a QuantyDB file next to the store's, <data>.raft (option a of 21 October). It is the simplest separation, and it costs a second fsync per write: one for the log, one when the command is applied to the store. Writing the log into the store's own file, so that applying and remembering the log share one commit (option b), waits for the measurement on the rented servers.

Decided on 26 October: option a stays for 17 November. It is built, runs in CI under SIGKILL and snapshots, and keeps the store's file usable on its own. Option b is the optimisation to try once the servers show where a write's time goes; etcd itself keeps a WAL apart from its data file. The price: on a cluster every write costs two fsyncs, which the talk's numbers say.

In CI three nodes run on one machine against the real etcdctl and an unmodified kube-apiserver given all three endpoints: writes through every node, reads and a watch through another, the same revision on every node, the leader killed with SIGKILL while the others go on, and the killed node coming back and catching up.

A node cuts its Raft log once 10,000 entries are applied beyond the last cut, keeping the last 5,000, as etcd's snapshot count does. A follower that needs more than the leader's log still holds gets a snapshot: the leader's live keys and metadata as of its last applied command, without history, in one message. The follower replaces its store with it, ends every watch on it as compacted so that its clients list again, and reads before the snapshot's revision are refused as compacted there; history on that node begins at the snapshot.

What it does not do yet, and what that costs: a snapshot travels as one frame of at most 256 MiB, so a larger store cannot bring a follower up yet. A command passed to a leader that dies before appending it is lost, and the client sees a timeout and sends it again, which etcd clients do. The store's history timestamps are each node's own clock at apply time, not the command's, so --ago can differ by milliseconds between nodes.