QuantyKube

ADR-033 Store nodes talk over TLS in frames, not gRPC, with certificates from a peer CA

All pages

ADR-033, from docs/DECISIONS.md in the QuantyKube repository.

Status In force

Raft messages between store nodes go over one TLS connection to each peer, as protobuf messages each prefixed with its length (src/raft/transport.rs), on a port of their own. Both ends show a certificate, and the peers' certificates come from a CA of their own, the peer CA, as etcd separates its peer certificates from its client certificates.

gRPC was the obvious choice and was rejected for now: QuantyCore's gRPC client makes one unary call per connection, and Raft wants a stream that stays open with messages flowing both ways. Frames on a TLS connection are a few dozen lines and give exactly that. If QuantyCore gets a streaming client, the frames can become a gRPC stream without the protocol noticing.

A sender drops what is queued for a peer while the peer cannot be reached, and a full queue drops too: Raft sends again, and a stale message reaching a peer that came back is worse than none.

A follower that gets a write from a client passes the command to its leader inside Raft, as etcd does, and answers the client when it has applied the command itself; a node answers reads after a ReadIndex from its leader. Every node applies the same commands in the same order, so every node computes the same answer.

The cost: a separate peer CA is one more thing to set up, and a client certificate from it is a key to the cluster's log, not only to its data. Anyone with a client certificate from the client CA must not also hold one from the peer CA. Using one CA for both, which small setups will want, makes every etcd client able to speak Raft.