ADR-015, from docs/DECISIONS.md in the QuantyKube repository.
Status In force
A soak run, a k3s control plane on QuantyKube under steady writes for
eight minutes, showed the server's memory growing by about 7 MB a minute
and not stopping: 18 MB to 74 MB. The store alone, driven in a loop with
retention, stayed flat at 7 MB, so the growth was in the server around
it. Rerunning with MALLOC_ARENA_MAX=2 cut it to a curve that levelled
off near 26 MB.
The cause is glibc's allocator, not a leak in the code: it gives
contending threads arenas of their own, up to eight per core, and memory
freed into one arena is not reused by a thread allocating from another.
A gRPC call is read on one thread, handled on a pool thread and answered
on a third, so memory moves between arenas all day. The server now caps
glibc at two arenas through mallopt at startup, from the syscall crate
(ADR-013).
The cost: some contention inside malloc under heavy parallel load, which the benchmarks will show if it matters, and a dependency on glibc's behavior that a musl build would not share.