ADR-031 Over a network the store speaks TLS and wants a client certificate

Supersedes ADR-004 where the store is reached over a network; the Unix socket stays as it was.

quantykube serve --listen serves the same etcd API over TCP with QuantyCore's TLS, as etcd does on port 2379. Every client must present a certificate from the CA given with --client-ca; a connection without one ends in the handshake. On the node the socket's mode was the only access control (ADR-004), and over a network the certificate takes its place: whoever holds one can read and write every object, which is what an etcd client certificate means too. kube-apiserver connects with its usual --etcd-cafile, --etcd-certfile and --etcd-keyfile.

Raft between store nodes (ADR-027), the API server on another machine, and the installer (ADR-009) all need this. In CI an unmodified kube-apiserver runs against the store over TLS, and etcdctl reads, writes and watches over it, and is refused without a certificate.

The gRPC server reads a connection on one thread and writes on another. The TLS state of a connection is shared between them behind a lock that each holds only while it moves bytes through it, never while it waits on the network.

Rejected: a token or password on top of TLS, which etcd's clients support but the API server does not use for etcd.

The cost: the TLS is QuantyCore's and has not been reviewed by anyone who did not write it (QuantyCore ADR-003), and it is now the lock on the whole cluster's data whenever the store is reached over a network. Until the review, a store reachable over a network should also be fenced in by the network itself.

Added on 21 October: a client that speaks HTTP/1.1 gets /version and /health as etcd answers them, and a client that starts with the HTTP/2 preface gets gRPC. The first bytes decide, not the handshake: kubeadm offers h2 there and then speaks HTTP/1.1. Nothing else is served over HTTP/1.1, and the client certificate is required the same way.