How I Redesigned the Credential-Fetch Client for a 9,000-Server Fleet

The problem

The last post on this topic argued for shipping the client as a single statically-linked Go binary. That was the packaging decision. It is not, on its own, an architecture. A credential-fetch client on a 9,000-server fleet does more than one HTTP call, it has to hand credentials to whatever local caller wants them, cache them so a cron-triggered process at :00 of the hour doesn't stampede the upstream vault, keep the cache honest under time-to-live pressure, survive an upstream outage without taking every batch job down with it, and produce an audit trail somebody can defend in a compliance review.

At fleet scale, the interesting architecture is not the call path, the call path is one HTTP request. It is what surrounds the call path. This post is the companion to the dependency-free-Go one; that post was about the shipping unit, this one is about the machine that shipping unit runs.

Prior art

Before writing anything, I looked at what the industry already ships for this shape of problem. Four options came up. None of them fit, but each one taught me something.

HashiCorp Vault Agent. The kitchen-sink solution. Templating, auto-auth, caching, response-wrapping, sinks, a lot of what I needed, plus a lot I didn't. The trouble was footprint and blast radius. A Vault Agent process sits around consuming tens of megabytes of resident memory; at 9,000 hosts that is a fleet-level number I did not want to spend on a client whose steady state is "sleep, wake up on socket connect, hand over a cached token, sleep again." Its dependency graph is also the entire Vault codebase, reasonable for a Vault-native shop, expensive to audit for a fleet where Vault is one of several upstream systems.

Custom Python daemons (the predecessor). A per-host Python process, requests for HTTP, a pickle cache on disk, a systemd unit that respawned it. It worked for years, and I have real affection for it, but the previous post covers the failure modes, pinned-interpreter drift, pip-mirror reachability, transitive-CVE surface, and a per-host process that carried its own OpenSSL and its own trust store, both of which had opinions.

Kubernetes service-account tokens with a projected volume. In principle beautiful, the kubelet mints a short-lived token, mounts it into the pod, refreshes it before expiry, workload reads a file. In practice, not applicable. The fleet is bare-metal RHEL. There is no kubelet. The projected-volume model assumes an orchestrator that owns identity and a filesystem lifecycle that gets torn down with the workload.

SPIFFE/SPIRE. A more general answer to the same question, workload identity via short-lived SVIDs, a per-node agent, a Workload API over a Unix socket. Architecturally the closest analogue to what I ended up building, and I borrowed the local-socket idea from it. It fell out on operational fit: SPIRE assumes you are ready to make workload identity a first-class platform primitive across teams, and we were solving a narrower problem, one credential type, one upstream, one caller shape. SPIRE would have been the right answer to a much larger question than the one I actually had.

The scoreboard, roughly:

The point is not that any of these are bad. Vault Agent and SPIRE are both good pieces of software. Neither of them was going to earn its keep on 9,000 hosts doing one call.

What we did differently

Three decisions distinguished the client from the prior-art baseline.

One shipping unit, no host assumptions. The previous post covers this, a single statically-linked Go binary, embedded CA bundle, no host trust store, no interpreter, no dynamic linker. Vault Agent and SPIRE both assume a runtime environment; we did not want one.

A local socket for callers, not a shared library or a subprocess-per-call. Callers are a mix of long-running services and cron-triggered short-lived scripts. A shared library would have forced every caller to link against it (blast radius: the fleet). A subprocess-per-call would have paid the cold-start cost on every credential fetch, for a cron script waking at :00, that is the entire runtime cost. A Unix domain socket at a well-known path meant callers open a connection, send a request, get a token, close, and the daemon amortizes everything expensive. The socket idea is borrowed from SPIRE. The rest is not.

The vault is authoritative but not on the hot path. Prior-art options treat the upstream vault as source of truth on every call, and pay a network round-trip to prove it. That is fine at low fanout and unacceptable at 9,000 hosts with cron-heavy callers. The client keeps a local encrypted cache with a conservative TTL, serves from cache when fresh, and refreshes asynchronously. The upstream vault sees a small, steady load instead of a synchronized :00 spike. That last decision is the one that most changes the operational shape.

System design

Eight components. Each one is boring on its own, which is the point.

(a) Local socket for callers. A Unix domain socket at /run/credclient.sock, permissions 0660, group-owned by a well-known local group that authorized caller services are members of. The wire protocol is a length-prefixed JSON request-response; no gRPC, no protobuf toolchain, no HTTP-over-UDS. The daemon spawns one goroutine per connection, reads one request, writes one response, closes. Median in-process response time on a warm cache is around 1-2 ms. This fits the workload because callers are simple, they want one token, now, without a client library, and because a Unix socket already gives us local-only authentication via filesystem permissions.

(b) Local encrypted cache with time-to-live. A single-file cache under /var/lib/credclient/cache.bin, AES-256-GCM at rest with a key derived from a host-bound seed (the machine-id plus a build-time salt, hashed once at startup, not a substitute for full-disk encryption, but enough to keep casual on-host reads honest). Entries carry a not-before, a not-after, and the credential blob. TTL is set well below the credential's real expiry, deliberately conservative, so the cache always refreshes before an entry actually goes stale. The file exists so the daemon can restart without a cold miss on the entire host.

(c) Async refresh vs sync-on-demand. The default path is async. A background loop wakes on a jittered interval (deliberately off-round to avoid synchronizing with cron), checks which cached entries are within a refresh window, and refreshes them out-of-band. Callers hitting the socket almost always get a warm cache. The sync-on-demand path exists as a fallback: if a caller asks for a credential the cache does not have (first boot, new caller identity, cache-eviction race), the daemon does the upstream fetch inline. The two paths share the same fetch code; the difference is who is waiting.

(d) Fallback strategy on upstream vault outage. The interesting one. When the upstream fetch fails, the daemon does not immediately fail the caller. It consults the cache: if the entry is expired but within a short grace window, the daemon serves the stale credential and marks the response with a stale=true flag, and emits an audit event. Past the grace window, the daemon refuses. This trades a small window of stale-but-valid credentials for a large reduction in blast radius during a vault outage, the alternative is that a 5-minute vault blip takes down every cron job on the fleet, which is the exact fleet-wide event we were trying to avoid.

(e) Audit trail emission. Every credential handoff writes a structured JSON line to stdout, which systemd's journal collects and a fleet log-forwarder ships to the central audit sink. The line carries a request id, caller uid, caller comm, credential group, cache-hit/miss, stale flag, and a truncated hash of the credential id (never the credential itself). No log framework, no rotation logic in the daemon; the journal handles it.

(f) systemd unit, not supervisord. Supervisord's advantage is that it works the same on every OS; systemd's advantage is that it is already installed on every host in the fleet. On a bare-metal RHEL fleet where systemd is the init system anyway, adding supervisord is adding a dependency to solve a problem systemd already solves. The unit is a Type=notify service with sd_notify from Go to signal ready, Restart=on-failure with a reasonable backoff, NoNewPrivileges, ProtectSystem=strict, ProtectHome=true, and a ReadWritePaths narrowed to the socket and cache directories.

(g) Rollout: Ansible playbook plus canary batching. The fleet's config-management is Ansible. The playbook does three things: drop the new binary at a versioned path, atomically flip a symlink, and restart the unit. Ansible's serial: lets you batch, the playbook runs in three passes with 24-hour soak windows between them. Canary batch: 90 servers, hand-picked to cover the long tail (old kernels, tight egress, unusual clocks). Second batch: 900 servers, still weighted toward diversity. Third batch: the remaining ~8,000. If any batch trips an alert (unit failure rate, socket-error rate on callers, upstream 5xx rate), the playbook halts and the on-call decides whether to roll forward or roll back.

(h) Rollback path. The versioned binary path exists exactly for this. Every rollout leaves the previous binary on disk. Rollback is: flip the symlink back, restart the unit, the same three Ansible steps in reverse, no re-download. In the worst case the rollback playbook runs against all 9,000 hosts and takes about as long as a normal Ansible push, which was well inside our tolerance for a bad-day scenario. We used it exactly once, for the failure mode below.

Numbers, since the brief asked for them: resident memory on the idle daemon runs around 6 MB on RHEL 8; cold start from process launch to socket up is about 15 ms (the same number the prior post cited for the binary itself); a warm-cache response over the socket is 1-2 ms and dominated by socket setup rather than lookup; a cold-cache response is 200-400 ms, upstream-latency-bound. Upstream vault QPS dropped by roughly an order of magnitude versus the previous synchronous client, visible in the vault's own dashboards the day the second rollout batch went out, though I did not run a controlled before/after benchmark and would rather flag that as directional than quote a precise number I cannot defend.

What broke first

The canary batch, day one. Ninety servers, and eleven of them started throwing socket-permission errors from a caller service. The service could not open /run/credclient.sock, permission denied.

The bug was not in the daemon. The daemon was creating the socket with the right permissions and the right group. The bug was in an assumption I had made about how a specific caller service was launched. On most hosts, that service ran under a systemd unit that put the process into the well-known group. On the eleven affected hosts, an older cron entry that did not know about the group launched the service. Same binary, same daemon, different launch path for the caller.

The fix was two things at once. Short-term: I widened the socket group to include a second, historically-present group, so the older cron-launched callers could connect. Long-term: I added a check-mode play to the Ansible rollout that surveyed each host for known caller processes and reported their supplementary groups, so the "what groups does the caller actually run under" question stopped being an assumption and became a piece of data.

The lesson was not "test permissions better." The lesson was that a 9,000-host fleet contains at least one of every historical launch pattern anybody has ever used, and the canary batch has to be picked to include those patterns, not to exclude them. The eleven affected hosts were exactly the kind of long-tail machines the canary was there to find. The system worked; the bug was cheap; the fleet-wide push did not happen with the bug in it.

What I would do differently

Three items for v2.

1. Single-writer discipline on the cache, enforced by design. Today the cache is single-writer because there is one daemon per host. It is not single-writer because the file format makes it so. If some future version ever runs two processes during a reload transition, the cache format does not defend itself. I would move to an append-only log with a compacting reader.

2. A proper telemetry channel that is not stdout. The audit trail via journald works and is defensible. It is not queryable at fleet scale without going through the central log sink, which adds minutes of lag. For fleet-level health, daemon uptime, cache-hit rate, stale-serve rate, upstream-fetch p95, I would ship a small metrics emitter to a fleet-metrics sink, on a separate path from the audit trail.

3. Cache-warming on rollout, not on first request. After a rollout, the daemon starts with an empty cache and warms as callers arrive. At the instant of a rollout, we shift traffic from cache-served to upstream-served for a brief window on every host, 9,000 hosts synchronized on a rollout schedule is exactly the thundering-herd shape the whole architecture exists to avoid. I would have the new daemon read the previous cache file at boot, validate signatures, and start warm.

Closing

The transferable lesson is that at fleet scale, the interesting architecture is almost never the call path. It is what surrounds it, where you cache, how you refresh, what you do when the upstream is down, who is allowed to talk to you, how you prove after the fact who asked for what, and how you push a change without turning every host into a coordinated failure. Prior art gives you components; the workload gives you the shape they need to fit. The bit that mattered here was not choosing Go, and it was not choosing a Unix socket. It was picking a shape where the vault is authoritative but not on the hot path, and then rolling that shape out slowly enough to let the long tail of hosts find the bugs before the bulk of the fleet did.

See also

• A Dependency-Free Go Binary Is the Right Answer for a 9,000-Server Fleet, the packaging decision this post assumes. Read that one first if the "why a static Go binary?" question is still open.
• How I Shrank a Multi-Cloud Telemetry Pipeline From 76 Minutes to 13, same fleet-ops thundering-herd shape at :00 of the hour, on a totally different workload.
• CipherStack's LRU Key Rotation, a small credential-vending service where the same "vault authoritative, cache on the caller" trade-off shows up on a much smaller fleet.

More on the platform work behind this, mutual-TLS credential fanout, fleet rollouts, and the security posture that motivated it, is on the projects page.