The problem
Madhya Pradesh Police, through the Smart India Hackathon 2022 problem statement, needed a way to take a seized hard drive or phone dump and turn every file on it, PDFs, scanned court paperwork, WhatsApp voice notes, bodycam clips, receipts photographed on a beat officer's phone, into something searchable inside an ongoing investigation. The workflow was not "upload to the cloud and wait." It was "an investigating officer, in a station, on an air-gapped laptop, needs to find the word Rajesh across 1,200 files in the next ten minutes."
That framing decided almost every architectural choice. It ruled out cloud OCR APIs. It ruled out any pipeline that assumed a beefy machine or a warm cloud runtime. And it forced us to reason about each modality separately, because the failure modes of a printed FIR are nothing like the failure modes of a shaky phone photo of an Aadhaar card, and pretending one model handles both was where every commercial demo we looked at fell apart on real seized data.
This post is about that whole pipeline, the per-modality preprocessing, the batching that fed the feature and index stages, and the unified index the officer actually queried.
Prior art
Before writing any of this, we spent about a week benchmarking the obvious things.
Google Vision / Document AI. Gold standard for print OCR quality in 2022, ahead of every open-source alternative on messy scans. Per-page pricing, cold-start latency in the hundreds of milliseconds, and, the hard blocker, cloud. Seized-evidence data on an ongoing case does not leave a police server. That closed the conversation before accuracy comparisons mattered.
Azure Document Intelligence (then "Form Recognizer"). Best-in-class on structured forms, the pre-built ID and receipt models were impressive against Indian government forms. Same cloud constraint. Per-page cost at our projected volumes was not a number a state IT budget was set up to absorb.
AWS Textract. Same shape as Azure. Strong tables and forms extraction, same cloud problem, and at the time its handwritten-text accuracy on Indian scripts was well behind its Latin-script accuracy.
Nanonets / Rossum. Enterprise document-AI vendors. Both had a compelling structured-extraction story with an in-the-loop training workflow a records clerk could operate. Both cloud-first, both per-document priced, and neither had a serious answer for audio and video.
Tesseract. The open-source baseline. Free, offline, battle-tested. Meaningfully worse than newer neural OCRs on anything not clean printed text at 300 DPI. Great as a fallback, not sufficient as the whole pipeline.
Every commercial offering wanted to be a horizontal document-AI product, one model per document type, per-page pricing, cloud. Our workload was vertical: one officer, one hard drive, all modalities, offline, one query surface. Nothing off the shelf fit.
What we did differently
Three decisions fell out of that gap.
Per-modality preprocessing, not one universal extractor. Every commercial system tried to be a general-purpose "give me a file, I'll return text." We split the pipeline by modality up front. A PDF took a different code path than a receipt photo, and a receipt photo took a different code path than a bodycam frame. Each path used the OCR/ASR family strongest for that modality on public benchmarks, even when that meant carrying four OCR models on disk.
Batch across files, not across modalities. The feature and index stages only ever saw one thing: batches of tokenized text. Everything modality-specific ran upstream, produced plain UTF-8 strings, and got fed into a length-bucketed batcher that did not know or care whether a transcript came from a Whisper pass or a Tesseract pass.
Unified TF-IDF index, not per-modality indexes with a federated query. Every "multi-modal search" paper wanted to embed each modality separately and rank in the query. In 2022, before semantic search became the default in production, this was overkill for a lexical query workload. Investigators searched for names, phone numbers, and vehicle registration strings. TF-IDF over the merged transcripts with modality metadata as extra tokens was the right tool. I will come back to why we did not do semantic embeddings.
System design
The pipeline had six real components. Ingest, per-modality preprocessing, tokenization and metadata merge, batcher, index build, and query router. Below is what each of them was and why.
1. Ingest
Files came in via a watched folder on the station's evidence workstation. Ingest classified each file by MIME sniffing (not extension, investigators renamed files) and enqueued it onto one of seven per-modality queues: born-digital PDF, scanned PDF, printed image, handwritten image, ID/form image, wild-scene image, audio, video. A single file could appear on multiple queues, a PDF with a mix of born-digital and scanned pages was split at page level before enqueueing. Queues were bounded (256 items) and backed by SQLite for crash recovery.
2. Per-modality preprocessing
This is where the modality-specific model choices lived. For each of the seven queues:
Born-digital PDFs (pdfplumber first, pdfminer.six fallback). Any PDF where the text layer was already present got its text extracted directly, in milliseconds per page, with no OCR. pdfplumber was primary because its table-aware extraction was noticeably better on FIR-style forms. pdfminer.six was the fallback when pdfplumber returned empty or threw on a malformed content stream, which happened often enough on scanned-then-OCR-embedded PDFs to be mandatory. This step alone handled roughly a third of the corpus by file count without ever touching an OCR model. Directionally: PDF text extraction was on the order of hundreds of times faster than the OCR path per page.
Scanned PDFs and printed images (PaddleOCR). For the neural OCR path on clean printed text, we ran PaddleOCR. The tradeoff triangle: Tesseract was fast, offline-ready, and weak on anything below ~250 DPI or with any skew. DocTR (Mindee) had the cleanest Python API and best layout preservation, but its Indic-language support was thin. PaddleOCR won because it shipped pre-trained models for both Latin and multiple Indic scripts, its inference throughput kept up with the batcher, and its detection stage handled slight skew without a separate deskew pass. It was the only one we could point at a Hindi-only scanned document and get usable output on.
Handwritten and messy documents (TrOCR + LayoutLMv3). TrOCR, Microsoft's transformer OCR, was the only open-source model we could get to do meaningful work on Indian carbon-copy forms and cursive handwriting. It is slow. It is model-per-word rather than per-page. We paired it with LayoutLMv3 for the subset of handwritten pages that were structured, forms, tables, statement pages, so we recovered which handwritten span was in which field, along with the raw text. LayoutLMv3 was the specialized model; TrOCR did the recognition. Known limitation: throughput. We routed as few files to it as possible.
IDs, receipts, and structured forms (LayoutLM family + Donut). For ID cards (Aadhaar, PAN, voter ID, driving licence), receipts, and structured forms, we used a two-tier approach. If the document classifier was confident about the form type, we ran Donut, the end-to-end OCR-free document-understanding transformer, because it went straight from image to structured JSON in one forward pass. When the classifier was unsure, we fell back to LayoutLM with an upstream PaddleOCR pass. Donut's limitation: on unfamiliar forms it produced confident-but-wrong JSON, which is worse than an error. The confidence gate caught that failure mode.
Wild-scene images (EasyOCR + PaddleOCR-Wild). Bodycam frames and phone photos are not documents. They are scenes with text somewhere in them, a signboard, a licence plate, a shopfront hoarding, a piece of paper on a table at an angle. EasyOCR was primary because its detection stage was tuned for arbitrary-orientation text in natural scenes. PaddleOCR-Wild was a second pass on frames where EasyOCR returned nothing. Neither was great. Wild-scene OCR was, and remains, the weakest link in this class of system.
Audio (Wav2Vec2, in 2022). The pipeline was built before Whisper. Whisper landed in late September 2022, right as we were finalising the SIH submission. The ASR of record was Facebook's Wav2Vec2, fine-tuned on Hindi and English via AI4Bharat and Hugging Face community checkpoints. Solid on clean speech, visibly weak on the phone-call recordings that were the majority of our audio corpus. The Samsung PRISM continuation in 2023 was where Whisper replaced it, a bigger accuracy win than any upstream batching optimisation. Today Whisper would be the default from day one, probably faster-whisper for the offline throughput profile.
Video (frame extraction + subtitle track + ASR fallback). Video was decomposed into three streams. Subtitle tracks were extracted with ffmpeg and merged as text. Frames were sampled at one per second (adjustable per case) and routed to the wild-scene OCR path, because visible text in a bodycam clip was often the whole point. The audio track was extracted and routed to the ASR queue as if it were a standalone audio file. Dedup happened at index time, if subtitle and ASR passes produced overlapping content, we kept the subtitle text.
3. Tokenization and metadata merge
Everything upstream produced a (file_id, modality, page_or_offset, text) tuple. This step normalised the text (Unicode NFC, whitespace collapse), tokenized with a Hugging Face tokenizer, and merged in a small set of metadata tokens, file path, modality tag, timestamp, as prefix tokens on the token stream. Metadata was tokenized as ordinary text so it fell into the same TF-IDF space as content; an investigator searching for bodycam or whatsapp would hit those files by modality even without content overlap.
4. Batcher
The short version: length-bucket the token streams into (256, 512, 1024, 2048, 4096) buckets so the feature and index stages always saw uniform batches, regardless of whether the upstream text came from a one-line receipt or a 400-page case file. Bucketing by length instead of count kept memory usage predictable on station-class hardware, which mattered because the machine was also the officer's working laptop, not a dedicated box.
5. Index build (unified TF-IDF)
Every modality's transcript, plus its metadata tokens, was concatenated into a single logical document per file and indexed with TF-IDF. IDF weights were computed over the merged corpus, not per-modality, a term rare across the whole corpus got a high weight regardless of whether it appeared in a PDF or a Whisper transcript. This is not what a semantic-search paper would recommend, and not what I would build in 2026. But in 2022, on a workload where queries were dominated by lexical matches, names, ID numbers, vehicle registrations, phone numbers, place names, TF-IDF was the right primitive. Cheap to build, cheap to query, easy to explain ("why did this file rank first? because Rajesh appears in it five times and nowhere else"). Explainability mattered more than we expected during trials, because the officer had to defend the ranking to a supervisor.
6. Query router
The query router did three small things and no more. It expanded transliterations (राजेश ↔ Rajesh) via a lookup table hand-curated from the corpus, it appended modality filters if the query used a modality keyword prefix (audio: Rajesh returned only files whose audio transcripts matched), and it ranked results using TF-IDF cosine similarity. No query-time reranker, no learn-to-rank model, no cross-encoder. There was no time and no data to train one, and the query volume was low enough that the lexical baseline was inside the SLO.
What broke first
The first thing to break in production, at the SIH final round, was the PDF path. A single 400-page malformed PDF, a scanned bundle of case papers where the text layer was garbage embedded by an old OCR, passed pdfplumber's "does the text layer exist" check with flying colours and returned pages of �������� where the text should have been. Because the check passed, the file skipped the OCR path entirely. Because the extracted text was gibberish, it also poisoned the TF-IDF vocabulary, a run of Unicode replacement characters became a "term" with high document frequency in that one file and zero elsewhere, which is exactly the shape TF-IDF weights heavily.
The fix was a lightweight sanity check after PDF text extraction: if the extracted text had fewer than a threshold fraction of characters in the expected script's Unicode block, we treated the text layer as absent and routed the file to the OCR path anyway. Two lines of code. But we shipped without it, and it took a demo failure to notice. Lesson: a pipeline with a fast path and a correct path needs a validator on the fast path, in addition to a router.
What I would do differently
Three things, in priority order.
Semantic embeddings alongside TF-IDF. In 2022 this was a defensible omission; in 2026 it is not. A dense retriever, sentence-transformers or an E5-family model, running in parallel with the TF-IDF index, with reciprocal-rank fusion at query time, would close the recall gap on paraphrased queries. The investigator who searches for vehicle and misses the file that says Bajaj Chetak is the case TF-IDF cannot solve without a hand-curated synonym table.
Whisper from day one. As above. Wav2Vec2 was the right choice given the timing; today the audio path should be faster-whisper in an offline configuration, and the frame-level dedup logic against the subtitle track survives the swap unchanged.
A page-level classifier upstream of the OCR router. We routed at file level, one file, one modality queue. But real PDFs are heterogeneous. A 200-page case file might have 40 pages of clean print, 15 of handwriting, and a scanned Aadhaar at the back. Routing each page independently would have let us keep the fast path fast and pay the TrOCR cost only on the pages that needed it. We ran a coarse version, born-digital vs scanned, but a proper printed / handwritten / form / photo classifier at page level was the shape the system wanted and did not have.
The transferable lesson
The thing that survived from this project into everything I have built since is the discipline of splitting a heterogeneous input space by its own natural boundaries before choosing a single downstream primitive. Every commercial vendor at the time was selling one model that "handles everything." What actually worked was to accept that different modalities had different failure modes, choose the best-in-class open-source model for each, and merge the outputs at the token layer where downstream stages could stop caring which modality a given piece of text came from. The index never had to know. Only ingest had to know, and that is where the complexity belongs.
See also
- HNSW or IVF-PQ? What I Actually Chose at 2M Documents, the semantic-search follow-on that the TF-IDF index in this post did not yet do.
More on the Multimedia File Indexer, Samsung PRISM 2023 Excellence Award, Smart India Hackathon 2022 winner adopted by MP Police, is on the projects page.