QuantyKube

ADR-015 Two malloc arenas for the server

All pages

ADR-015, from docs/DECISIONS.md in the QuantyKube repository.

Status In force

A soak run, a k3s control plane on QuantyKube under steady writes for eight minutes, showed the server's memory growing by about 7 MB a minute and not stopping: 18 MB to 74 MB. The store alone, driven in a loop with retention, stayed flat at 7 MB, so the growth was in the server around it. Rerunning with MALLOC_ARENA_MAX=2 cut it to a curve that levelled off near 26 MB.

The cause is glibc's allocator, not a leak in the code: it gives contending threads arenas of their own, up to eight per core, and memory freed into one arena is not reused by a thread allocating from another. A gRPC call is read on one thread, handled on a pool thread and answered on a third, so memory moves between arenas all day. The server now caps glibc at two arenas through mallopt at startup, from the syscall crate (ADR-013).

The cost: some contention inside malloc under heavy parallel load, which the benchmarks will show if it matters, and a dependency on glibc's behavior that a musl build would not share.