What the problem actually was
The HackerRank Orchestrate hackathon on May 1-2, 2026 handed us a CSV of support tickets spanning three product surfaces: HackerRank, Claude (Anthropic's assistant), and a Visa help-desk corpus. The task was one CSV in, one CSV out. For every ticket, my agent had to decide five things: status (replied or escalated), product_area, the customer-facing response, a justification, and a request_type bucket.
24 hours. One submission. A local Anthropic-compatible proxy at localhost:6655 for the LLM calls (I assume the organisers provided it, but the repo doesn't say). A knowledge base of 774 markdown files: 438 HackerRank docs, 322 Claude docs, 14 Visa docs. The README rounds that to "about 800," which is close enough.
Most of my time did not go into the retrieval stack. It went into what sits in front of it and what sits behind it.
The retrieval stack
The retrieval layer is the least interesting part of this project, which is exactly why it worked. Hybrid BM25 + dense vectors, fused with Reciprocal Rank Fusion, top-10 out.
%%{init: {'theme':'default'}}%%
flowchart TD
Q[Incoming ticket] --> DR[Domain router: company name first, then keyword scores]
DR --> BM[BM25 top 20]
DR --> VEC[Vector top 20: BGE-large-en-v1.5, 1024d, L2 normalised]
BM --> RRF[Reciprocal Rank Fusion, k=60]
VEC --> RRF
RRF --> PRI[Source priors: FAQ x1.3, index.md x0.3, release-notes x0.5]
PRI --> TOP[Final top 10, RRF >= 0.015 gate]
TOP --> LLM[Claude sonnet 4, temp 0, few-shot prompt]The embedding path is a three-step cascade in embeddings.py: try the Hugging Face Inference API, fall back to a local ONNX runtime, fall back again to TF-IDF. Model fallbacks inside those steps: BAAI/bge-large-en-v1.5 -> bge-base -> bge-small. Every vector is L2-normalised to 1024 dimensions.
The knobs in config.py, mostly untouched during the 24 hours: BM25_TOP_K=20, VECTOR_TOP_K=20, RRF_K=60, FINAL_TOP_K=10, MAX_CHUNK_TOKENS=512, CHUNK_OVERLAP=50. An answerability threshold at RRF >= 0.015 gates whether the agent even tries to reply, versus falling through to escalation.
The one place I put opinions into retrieval was source priors. index.md files got a 0.3 weight (navigation, not answers). Release notes got 0.5. Applicant-tracking integration docs got 0.7 (a common false-positive on HackerRank queries). FAQ and troubleshooting got 1.3. Three files were skipped via SKIP_FILES: consumer.md, merchant.md, checkout-fees-contact-form.md. All noise-generators in early tests.
The fast path in front of it
RAG is expensive per query and it hallucinates in specific, predictable ways. So I put a rules layer in front, and it caught the four categories where retrieval-then-generation is worse than a two-line policy response.
detect_fast_path in agent.py runs four checks, in order:
- Gratitude. If the ticket is under 100 characters and matches
thank|thanks|thx|cheers|appreciate, reply "Happy to help" and close. Retrieval on a thank-you note is a waste of tokens and a chance for the model to volunteer information the customer didn't ask for. - Platform outage. A regex for "is it down / are your servers up / getting 500 errors" would false-positive on any ticket mentioning "my project is down" or "my API key isn't working," so this check runs an exclusion regex first ("my project", "my account", "bedrock", "api key" and a few more), then three outage patterns. If both pass, the ticket escalates without ever touching retrieval.
- Malicious code request.
rm -rf,drop.*database, "delete all files." The LLM will refuse anyway, but I'd rather return a clean "out of scope" than watch the model produce a paragraph of safety boilerplate. - Off-topic keyword list.
iron man|avengers|pokemon|game of thrones|netflix|spotify|instagram|tiktok|who is the president. Gated by a check that the ticket isn't also mentioning one of the supported products, so "our HackerRank integration for Spotify" doesn't get shot down.
Rules-then-RAG rather than RAG-alone was a bet that the tail of trivially-non-answerable queries is fat enough to be worth carving off. I don't have production numbers to prove that on this corpus. I do know that every one of these categories was something I saw in the sample tickets and did not want the LLM near.
The grounding verifier
The part I spent the most time on is a small function called verify_grounding that runs after the LLM produces its response, not before.
The problem: the model is happy to invent phone numbers, URLs, and support email addresses. It has read a lot of them during pre-training. When a ticket says "how do I contact enterprise support?" the model has an opinion, and that opinion may or may not be in the retrieved evidence.
The fix is not more prompting. It's regex extraction plus substring matching against the concatenated evidence text.
verify_grounding pulls three things out of the LLM output: URLs, phone numbers, and email addresses. It checks each extracted item against the evidence text that fed the generation. For phones, I match on the digit-stripped form, so +1 (800) 555-0100 and 18005550100 compare equal. If the item is not found in the evidence, the verifier replaces it inline with a sentinel: [link removed], [number removed], [email removed].
Two design choices worth naming.
After the LLM, not before. I could have instructed the model to only use URLs from the evidence. Instructions don't hold. A regex substring check does. The model can hallucinate freely; the verifier catches it before the response is emitted.
The 50% escape hatch. If more than half the extracted items get stripped, the verifier appends a note to the justification field explaining that the response was heavily filtered. This is a signal to a human reviewer without breaking the CSV contract. The response column stays customer-safe. The justification column is where the mess goes.
The trade is that this only catches URLs, phones, and emails. A hallucinated version number, a wrong policy claim, or a fabricated feature is invisible to it. Grounding is a spectrum. I picked the part of the spectrum where regex is a scalpel.
The commit history as a shipping signal
The hackerrank-orchestrate-RAG repo has 13 commits. Twelve of them are documentation and diagram polish: SVG redraws, arrow reroutes, "Fix cropped architecture diagram," "Switch architecture diagrams to inline Mermaid," "Add LinkedIn architecture card." Exactly one commit contains code: the initial commit, which dropped all nine Python modules (2,679 lines across agent.py, indexer.py, preprocessor.py, embeddings.py, prompts.py, retriever.py, main.py, llm.py, config.py) plus a tests/ directory in a single push.
This is what a hackathon shipping shape looks like from the outside. Design and prototype off-repo, iterate locally without polluting the log, land the whole system in one commit when it works, spend the last hours making the artefact presentable. Not how I'd ship a long-lived service. Exactly how I'd ship a 24-hour submission where the deliverable is one CSV and one repo the judges will glance at.
One thing the commit history hides: there's a dead-code smell in agent.py inside _build_few_shot_messages. user_msg gets assigned once from a ternary that breaks the f-string chain, then reassigned immediately below with a comment saying "Fix: build user_msg properly." The first assignment is overwritten. It works. It's ugly. The kind of thing that survives a 24-hour build and doesn't survive a code review.
What I don't know
Being honest about the unknowns is part of writing this up.
I don't know how the agent scored. Results were promised for May 15, 2026, and I do not have submission metrics, a leaderboard position, or an output CSV committed to the repo. There was a judge interview stage; I don't know how that landed.
I don't know whether the 67 tests in code/README.md still pass on today's dependency set. The requirements.txt is pinned (anthropic==0.79.0, rank-bm25==0.2.2, onnxruntime==1.24.2, Python 3.14+), which reduces drift, but I haven't rerun them since submission.
I don't know how often the fast-path rules actually fired on the judging set. I know they fired in local testing. No hit-rate breakdown.
And I don't know whether the localhost:6655 Anthropic-compatible proxy is a HackerRank thing or something the judges rewired for their evaluation environment. The code assumes it's there. If it isn't, llm.py fails loudly at the first ticket.
What I would keep, what I would change
The rules-then-RAG-then-verifier shape is the part I would keep. Retrieval is a means, not an end. The interesting problems in a support-triage agent are at the two boundaries: what deserves retrieval at all, and what the LLM is allowed to say when retrieval is done.
The verifier is the piece I would generalise. Regex over URLs, phones, and emails is the easy tier. The next tier is entity extraction on the evidence and on the response, and rejecting response entities that don't appear in the evidence set. That's an extra model call per ticket, which for a hackathon budget was too expensive. For a real deployment I'd pay it.
The _build_few_shot_messages dead code would go first.
Repo: kaushiksaravanan/hackerrank-orchestrate-RAG. More projects on the projects page.