CipherStack Beyond the LRU: Full Vault Architecture

The problem

The LRU rotation post covers one mechanism, how CipherStack picks which of eight Gemini keys to hand you. That mechanism is important, but it is also the smallest, most testable box in the diagram. The rest is what makes it a vault instead of a Postgres table with a clever ORDER BY: encryption at rest, the auth surface, the deploy target, the dashboard, the metrics.

I kept putting off writing this because none of the interesting decisions are load-bearing on their own. They are load-bearing together. This is a post about the composition, not any one component.

Prior art

I did not build this from a blank page. Everyone who has ever done credentials at any scale has already picked a lane, so I looked at what was on the shelf first.

• HashiCorp Vault is the enterprise-grade answer. Dynamic secrets, transit encryption, PKI, an auth-method surface as wide as an aircraft carrier. It is also the heaviest tool in the category, you run a Raft cluster, you manage seal/unseal ceremonies, and every side project that wants a Gemini key pays for a five-node HA topology. For solo work it is overkill.
• Doppler is what I would recommend to a startup. It nails developer ergonomics, CLI, per-environment configs, real-time sync into deploy targets. The catch for me is that it is a hosted SaaS, and I did not want a third-party vendor sitting in the request path between my hackathon Discord bot and its Gemini key.
• AWS Secrets Manager and GCP Secret Manager are the boring correct answer if you are all-in on one cloud. What they do not do is treat a group of provider keys as a rotatable pool with LRU semantics, rotation is "generate a new value for this one secret," not "pick the least-recently-used key from a fleet of eight." And I run projects across four different targets, so leaning on one cloud's IAM was going to leave three of four out.
• Infisical is the OSS one closest to what I wanted. Self-hostable, has a CLI, has rotation. The rotation model is still "one secret, one value, updated on a schedule", the fleet primitive is not there, and the deployment shape was a Node monolith plus its own Postgres and Redis, which was more moving parts than I wanted for what is fundamentally a lookup table with two indexes.

None of these are wrong. They are all built for a workload that is not mine: one operator, twenty-ish side projects, ~200 keys across ~24 providers, and a specific need to treat multiple keys per provider as a rotating fleet.

What I did differently

Given that baseline, four architectural choices fell out.

Fleet-as-primitive, not secret-as-primitive. Every prior-art tool models a key as a scalar, one Gemini key, rotated on a schedule. CipherStack models the primitive as a group, eight Gemini keys, and vending returns whichever one is least-recently-used and not in cooldown.

Two auth paths, two threat models. Service tokens (long-lived bearer, for my own .env files) and certificates (HMAC-signed per-request, for anything I deploy publicly). Ergonomics for a project I trust differ from ergonomics for a Fly.io deploy that faces the internet.

No secret dump, ever. The API deliberately has no "list keys in group" endpoint. You can vend. You can report by ID. You cannot enumerate.

Small blast radius per component. No Redis, no message queue, no worker fleet. One Fly.io machine with Postgres attached. If a component fails, there is exactly one to look at.

System design

Here is the whole thing, top to bottom.

Postgres row, the storage layer

Every key is a row in a single api_keys table. Columns of interest: id, group_slug, provider, base_url, encrypted_key, encryption_nonce, status, last_vended_at, vend_count, cooldown_until, exhausted_until, input_tokens_total, output_tokens_total, created_at.

The encrypted_key column is a bytea containing an AEAD ciphertext produced by libsodium's crypto_secretbox, XChaCha20-Poly1305 with a 24-byte nonce. The 24-byte nonce is wide enough that generating one from /dev/urandom per encryption is safe without any counter or KDF hoops. The AEAD tag covers the ciphertext plus the row's id as associated data, so a swap-in attack (moving a ciphertext from row A into row B) fails at the auth-tag check.

The master key lives in a Fly.io secret and is loaded into process memory at boot. It never touches disk on the app side. Rotating the master key is a manual re-encrypt pass, I have done it once, in about 40 seconds against ~200 rows.

The table is small enough that indexes are theatre, but there is a btree(group_slug, last_vended_at) for the vend query and a partial index on cooldown_until. Both are cheap.

The API layer, Fly.io, one machine

The API is a Rust-flavored HTTP service (Axum, sqlx, libsodium-sys) running on a single Fly.io shared-cpu-1x machine with 512 MB of RAM, in iad because Postgres lives there too. No load balancer, no autoscaler, no worker pool. One machine, one process, one Postgres connection pool.

Why one machine? The whole vault does ~5,000 vends a day. That is one vend every 17 seconds on average. Autoscaling this would add a control plane whose failure modes are strictly worse than the failure mode of the machine it is protecting. If the machine dies, Fly restarts it in seconds; every client already has retry logic because every HTTP call has retry logic.

Endpoints, in full:

• POST /api/v1/vend/{group}, bearer-token, returns one key.
• GET /api/v1/cert/vend/{group}, cert-signed variant of the above.
• POST /api/v1/report, usage or rate-limit signal.
• GET /api/v1/groups, list groups (not keys).
• GET /metrics, Prometheus scrape endpoint.
• Everything under /dashboard/*, session-auth React app.

Six real endpoints. That is the whole API surface, and keeping it that small is a deliberate defense. The vault cannot leak what it does not expose.

The certificate handshake, headless auth

For anything I deploy publicly, the service token model is wrong. A token in a Fly.io secret is fine until the day I misconfigure a preview environment and the token ends up in a build log. What I wanted was an auth scheme where a leaked signature was only replayable for a small window, and where the credential itself never traveled over the wire.

The handshake is boring on purpose. Each certificate has a server-side secret. The client signs "{timestamp}:{group_slug}" with HMAC-SHA256 using that secret, sends the signature and timestamp as headers, and the server rebuilds the string and verifies. Timestamps outside a 5-minute window are rejected.

Two properties that matter. First, the secret is only shown to the user once at certificate creation, the server stores a hash. Second, the signature is scoped to the group being vended from, so capturing a signature for gemini does not let an attacker vend from elevenlabs. Scope is baked into the signed payload.

This is basically the AWS SigV4 pattern trimmed to a single header pair, but it is the piece of CipherStack I am most quietly proud of, because it lets me drop a cert into a public-ish deploy target without lying awake about the blast radius.

LRU rotation, the vend state machine

Covered in the previous post. One sentence for continuity: a four-state machine (available → in-flight → cooldown → exhausted) collapsed into a single UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED) statement, so the whole thing is one round-trip and concurrency-safe by construction.

Per-group usage reporting

Every vend returns a key_id. Clients that care about cost tracking can POST /api/v1/report with {key_id, input_tokens, output_tokens} after their downstream call completes. The vault increments a running counter on the row.

Nothing fancy, no time-series database, no OLAP rollup. Just monotonic counters on the same row that the vend query updates. If I ever want per-day rollups I will add a usage_daily table, but at 200 rows the raw counters are fast enough.

Reporting a 429 is the more interesting call, it flips cooldown_until = NOW() + 60s and lets the vend query naturally skip the row, turning a rate-limit signal from an out-of-band alerting problem into a normal in-band API call.

The dashboard, React app, session auth

The dashboard is a small React app served from the same Fly machine. Login is username + password (argon2id hash, session cookie, Secure + HttpOnly + SameSite=Lax). Once you are in, you can add keys, revoke keys, mint service tokens, mint certificates, and see per-group usage.

The one design decision worth calling out: the dashboard can reveal a plaintext key, because sometimes you actually do need to copy a key into a legacy config file, but doing so is gated on a re-auth prompt. That means a stolen session cookie cannot silently exfiltrate plaintexts, and most of the time I never see the plaintext of any of my own keys.

The /metrics endpoint, Prometheus

GET /metrics returns a Prometheus text-format payload. Counters for total vends per group, cooldown transitions, failed auth attempts, per-endpoint request counts, and histograms for vend latency. It gets scraped every 15 seconds by the same Grafana Cloud pipeline as the multi-cloud telemetry setup.

The specific reason I care: failed auth attempts. If someone brute-forces the cert handshake, the timestamp check will reject nearly all attempts, but the failure counter will spike, and I have an alert on it. It is the closest thing this system has to an IDS.

What Dyx actually calls it for

Concrete usage: the Dyx voicemail line is the busiest client. Every incoming call vends one Gemini key from the gemini group for the LLM turn, one ElevenLabs key from elevenlabs for TTS, and (if the ASR path falls back) one Groq key from groq for Whisper. Up to three vends per call. Directionally, this accounts for maybe 15% of the daily vend traffic; the rest is dashboards and cron jobs.

Dyx uses per-call vending rather than a cached key because a call is exactly the boundary where cooldowns matter, if the previous call rate-limited the Gemini key, the next call should get a different one. Caching would defeat the whole point.

Threat model, what this does and does not protect against

Being honest about a threat model is uncomfortable, and being dishonest about one is worse. What CipherStack actually protects against:

• Plaintext keys in git. Every key lives in exactly one place, encrypted.
• Passive database compromise. A snapshot of the Postgres volume is ciphertext. Without the master key (which is in a Fly secret, not on the volume), the rows are useless.
• Long-lived signature replay. The cert handshake caps replay windows at 5 minutes.
• Signature scope escalation. A signature valid for gemini cannot be repurposed for elevenlabs.
• Silent plaintext exfiltration from a stolen session. Dashboard reveals require re-auth.
• Rate-limit-driven service outage. LRU + cooldown means one exhausted key does not black-hole an entire provider.

What it explicitly does not protect against, and I want to be direct:

• Compromise of the running application process. If someone gets code execution on the Fly machine, they have the master key in process memory. There is no HSM. There is no split-key ceremony. I run one machine because I trust Fly's isolation more than I trust my ability to run a five-node Raft cluster correctly, but that trust is load-bearing.
• Compromise of the operator's browser session. Re-auth is a speed bump, not a wall.
• Compromise of Fly.io's secret store. The master key lives there.
• Nation-state adversaries. A determined well-resourced attacker has many easier paths than the vault itself (see: my laptop, my phone, my Google account).

If you need protection against any of the "does not" list, you need HashiCorp Vault plus an HSM plus a security team. CipherStack is designed for the threat model of "a solo operator who wants to stop leaking API keys and stop hardcoding rotation."

What broke first

The most memorable failure was not the vend race, that one I caught in dev, and the LRU post documents the fix. The one that made it to production was a nonce-reuse scare that turned out not to be one.

I had written the encryption helper to generate a 12-byte nonce (what AES-GCM wants). When I swapped the primitive to XChaCha20-Poly1305 partway through development, because I wanted the wider 24-byte nonce for safety margin, I updated the encrypt path but forgot to update one of the decrypt paths, which was still reading 12 bytes off the front of the ciphertext. It decrypted correctly for keys encrypted before the change (the old 12-byte nonces still fit) and failed for anything encrypted after. The failure surfaced when a newly-added Groq key returned 500 on vend.

The fix took ten minutes: bump the nonce read width, add a version byte to the ciphertext envelope so future primitive changes are self-identifying, backfill the version byte on existing rows. The lesson was less the specific bug and more the shape of it, I had unified my encrypt path and split my decrypt path, which is the wrong direction. Every subsequent AEAD change goes through a shared open() / seal() pair, with a test that round-trips every version tag through both.

What I would do differently

Three things I would revisit if I started this over.

1. Version the ciphertext envelope from day one. The nonce-reuse scare would have been a one-line fix with a version byte from the start. I retrofitted one; I would rather have designed it in. Any format that survives longer than a week wants a version tag.
2. Move cooldown state out of the main row. Every vend does an OR cooldown_until < NOW() check against the main table. Fine at 200 rows, fine at 2,000. But cooldowns are short-lived and don't need durability, losing them on restart would just mean a 60-second window of slightly-less-optimal LRU. A Redis sorted set is the right shape.
3. Structured audit log. The only durable record that key X was vended at time T is last_vended_at, which is overwritten on the next vend. For any post-incident forensics, that gap would be the first thing I regret. A tiny append-only vend_log table is the v2 fix.

Closing

The transferable lesson is smaller than it looks. It is not "encrypt your secrets" or "rotate your keys", everyone knows those. It is that most systems that call themselves vaults are actually two independent components in a trench coat: a storage substrate that guarantees confidentiality at rest, and a policy engine that decides who gets what, when, and under what constraints. Prior-art tools tend to nail the first and treat the second as configuration. CipherStack's whole thesis is that the interesting work lives in the policy engine, LRU, cooldown, per-group scope, cert-scoped signing, and the storage substrate is the boring, well-understood dependency. The crypto is the easy part. The policy is where you earn the label.

See also

• An LRU Key-Rotation State Machine for a Personal Credential Vault, the state-machine deep dive that this post is a companion to.
• Dyx: the sub-700ms latency budget for a personal AI voicemail line, the busiest CipherStack client, per-call token vending in the wild.
• From 76 to 13 minutes: multi-cloud telemetry consolidation, the same Grafana Cloud pipeline that scrapes /metrics.

CipherStack is live at cipherstack.kaushik.cv. Public docs and the LLM-friendly llms.txt.