AarogyaVaani: a voice-first ASHA didi that remembers who called yesterday

What AarogyaVaani actually is

aarogya is Sanskrit for well-being. vaani is voice. The full phrase reads as "the voice of health," which is roughly how the product presents itself to the caller. Someone in a village opens the web app, taps a big call button, and starts talking. The assistant picks up in Hindi, English, or Kannada, the three languages the current system prompt actually supports (the underlying language module carries codes for many more, but the didi persona is only tuned for these three today). The persona is an ASHA didi, the same accredited social health activist a rural family already knows from their block PHC.

I built this as a single-repo project under kaushiksaravanan/aarogyavaani. Two Vercel projects run off the same repo: aarogyavaani-api.vercel.app for the FastAPI backend and aarogyavaani-app.vercel.app for the Vite frontend. The real code lives on main (the master branch still has just the LICENSE and an initial commit, which is a mess I have not yet cleaned up).

The system prompt is the whole product

Voice-first health tools live or die on how the assistant talks. Vapi handles the loop (GPT-4o brain, Deepgram STT, ElevenLabs TTS), so the levers I had were the system prompt and the tool schemas. vapi_config/assistant_config.json is where most of the design ended up.

A few choices from that prompt that mattered more than any code I wrote:

• Formal address. The assistant always uses "aap," never "tu" or "tum." A didi from the health center would not tum a stranger.
• Village vocabulary. The instructions tell the model to say "sugar ki bimari" and not "diabetes mellitus," "BP badh gaya" instead of "hypertension." A caller who can name their condition can also refuse it.
• One question at a time, with a repeat-back for confirmation. Rural calls happen over 2G handsets with kids crying in the background. If the assistant asks three things at once, none of them get answered.
• A hardcoded emergency short-circuit. If the model detects chest pain, stroke signs, or severe bleeding, it must say "Yeh emergency hai! Abhi turant 108 ambulance ko call karein!" verbatim and stop the medical chat. That line is baked into the prompt so it never gets paraphrased away.
• A mental-health handoff to iCALL at 9152987821. When distress signals show up, the didi offers the number instead of trying to counsel through Vapi.

I keep going back to the prompt file more often than any Python module. It is the closest thing to a design doc that this repo has.

Memory: same phone, continuous chart

Every call ends with a summarisation step. The transcript, the tools the agent invoked, the medicines the caller mentioned, the follow-up dates: all of that gets condensed and written to Qdrant, keyed by the caller's phone number. On the next call from the same number, that context loads into the assistant's opening turn.

The effect is small and important. A grandmother who called on Monday about her father's sugar readings does not have to re-explain on Thursday. The didi can open the next call by referring back to what they already discussed. Same phone number, continuous chart.

There is no graph memory here. I want to be precise about that because my other projects have gone in the graph direction and someone will read the wrong thing. AarogyaVaani's memory is vector search over per-call summaries. Nothing more exotic than that.

The call flow in one picture

flowchart LR
  User[Caller on 2G phone] -- speech --> Vapi
  Vapi -- STT + LLM --> API[FastAPI on Vercel]
  API -- tool calls --> Agents[Multi-agent orchestrator]
  Agents -- RAG --> Qdrant[(Qdrant KB + memory)]
  Agents -- vision / text --> LLM[GPT-4o via OpenRouter, Gemini fallback]
  Qdrant --> Agents
  LLM --> Agents
  Agents --> API
  API -- structured reply --> Vapi
  Vapi -- TTS --> User
  API -. per-call summary .-> Memory[Qdrant memory index]
  Memory -. next call opens with context .-> Vapi

Fourteen tools, one orchestrator

aarogyavaani/app/agents.py is 1,190 lines. It exposes a fixed set of tools that the Vapi assistant can call mid-conversation:

search_knowledge, search_memory, get_medications, get_reports, assess_emergency, generate_tasks, get_doctor_brief, compare_reports, get_health_report, get_family_members, proactive_health_check, match_schemes, detect_family_context, smart_scan_info.

That covers the shape of the product. The caller can ask a question about diabetes and hit search_knowledge against a curated Qdrant index. They can upload a lab report; GPT-4o vision extracts medicines and readings; get_medications and compare_reports walk them through what changed. They can ask about Ayushman Bharat, JSY, or PMMVY and match_schemes returns eligibility. They can add a mother, a father-in-law, a child, and the assistant picks up which family member the current question is about.

The knowledge base under aarogyavaani/knowledge_base/ has English, Hindi, and Kannada content for diabetes, maternal health, and Ayushman Bharat. That is the ground truth the RAG tools search against, not the open web.

Free-tier economics via key rotation

The one piece of infrastructure I am proud of is aarogyavaani/app/keypool.py. Embedding models cost real money on OpenAI. Every free tier has small quotas and generous rate limits. So the pool holds keys across HuggingFace, Mistral, Cohere, and Nvidia inference endpoints and cycles through them.

HuggingFace alone has three keys in the pool. When one returns a 429, 401, or 403, the pool marks it in cooldown and picks the next one on the next request (commit 084eab4 is where the cycling landed). LLM calls have a similar pattern: OpenRouter for GPT-4o-mini as the primary, Gemini as the fallback (commit 76790aa added the fallback path after a nasty OpenRouter outage).

The math is not glamorous. A modest pool of keys across five providers keeps the app inside free-tier limits for a small pilot, and the caller never sees a "model unavailable" error because someone else already exhausted a quota.

Monorepo, two Vercel projects

aarogyavaani/vercel.json uses @vercel/python on app/main.py and ships to aarogyavaani-api.vercel.app. The frontend vercel.json under aarogyavaani/frontend/ runs the Vite build and ships to aarogyavaani-app.vercel.app. Same repo, two projects, one push.

The frontend is React 18, Vite 6, Tailwind 4, Clerk for auth, @vapi-ai/web for the call widget, pdfjs-dist and tesseract.js for on-device prescription parsing. Nineteen pages under src/pages/, of which the ones that get exercised most on demo calls are Call, Dashboard, History, DoctorBrief, Medications, Family, and SchemeMatcher. The Vapi public key is hardcoded as a fallback in the frontend (commit ab01c09) because the whole point is that a caller can land on the site and dial without setting up a keychain.

NurseAvatar.jsx is 445 lines of animated SVG. Volume-reactive eyes, occasional blinks, cursor tracking. It is a UI detail more than an engineering achievement, but it is the thing users mention on video calls, so it earns its LOC.

Deploy artifacts I keep around: capacitor.config.ts for the Android wrapper, a Dockerfile, render.yaml, Procfile, docker-compose.yml. Not all of these are shipping paths. The Capacitor wrapper is aspirational, and I have not yet installed the app on a physical device.

What is still unfinished

Being honest about the repo state matters more than looking clean.

• The root directory still carries scaffolding from launchforge, the SaaS-landing template I rebranded once (f215828) before the real work started at 9344848. The template files should be deleted; they still live at the root.
• Six Git Reporting Tool - Automated Reports...html files sit at the repo root. I did not put them there deliberately and I have not read them. Likely stray downloads.
• docs/hackathon-playbook.md exists but I have not opened it recently, so I cannot in good faith tell you which hackathon this was scoped for or what the deadline was.
• The two Vercel URLs may or may not be live at the moment you read this post. Vercel's free tier hibernates cold builds, and I have not set up an uptime check.
• The test branch has drift that I have not audited.

None of that is fatal. It is the shape of a project that grew from a template into a real thing over about twenty commits on main. I would rather ship the write-up now with the leftover template still on disk than delay it until the repo is spotless.

Why voice, and why now

Text chatbots for rural health assume a keyboard, a data plan, and enough literacy to type a symptom into a form. Voice with a phone number assumes almost nothing. The lift I got from putting an ASHA didi persona on top of an already-good voice stack (Vapi handles the parts I would have spent three weeks on) was larger than any model upgrade I could have chased. Most of what makes AarogyaVaani feel useful is that it sounds like someone the caller has met before, and it remembers what they said last time. Both of those are prompt and memory decisions, not model decisions.