Somewhere between September 2022 and August 2023 I made three repositories with sign language in the name. On paper they look like a plan: recognition, then generation, then a modality bridge from audio into sign. Reading the code back now, the plan is more of an after-the-fact story. What was really happening was a student trying to figure out where the hard part of accessibility ML actually lives.
The human problem is easy to state and hard to fix. A deaf or hard-of-hearing signer trying to talk to a hearing person who does not sign has two directions of failure. The signer's hands need to become text or speech. The hearing person's speech needs to become something the signer can read. Both directions are open research. And every serious deaf-community source I have read since says the same thing: the tech people usually solve the wrong version of the problem, and produce something no signer would actually use.
Repo one: camera to text with an LSTM
The first repo was Sign-language-Camera-to-Text-LSTM-Real-time, pushed in September 2022. Its ambition, on the tin, is real time camera to text. Its actual scope was smaller. I trained a classifier on three letters at a time (A, B, C in one run, D, E, F in another), 30 short clips per letter, 50 frames per clip. Each frame passed through MediaPipe Holistic to get 126 hand keypoint values, and the sequence of frames went into a three-layer LSTM (128, 64, 64 units) feeding a dense head. Sigmoid activations, categorical cross-entropy, an Adam optimizer, and a callback that stopped training once accuracy crossed 0.7 with loss under 0.3 after epoch 50.
Why LSTM in 2022, when transformers were already the default for sequence modelling? Honest answer: because I had done LSTMs in coursework and understood them, because the sequences were 50 frames of 126 numbers and did not need attention over long context, and because the training data I could collect on my own webcam was tiny. A transformer trained on 30 examples per class would overfit in a way I would not be able to diagnose. LSTMs on small keypoint sequences were the boring correct choice for a bedroom setup. The tradeoff is that even at 0.7 accuracy on my own hand, the model was really learning "Kaushik's A, in my lighting, in my apartment." That is not a sign language recogniser. That is a personal gesture recogniser dressed up in academic vocabulary.
The second thing I noticed reading it back is that the letters were fingerspelled letters, not signs. In ASL and Indian Sign Language, fingerspelling is a small part of communication, mostly reserved for names and loan words. Actual signed conversation uses whole-word signs, facial expressions, spatial grammar, and directional verbs. A three-letter fingerspelling classifier trained on 90 short clips is a computer vision demo. It is not a communication tool.
Repo two: the generation direction, sort of
Sign-language-generation was pushed in November 2022 and set to private. In my head, and in the title, this was going to be the second act: text goes in, sign comes out, either as a rendered signing avatar or as a sequence of pose skeletons.
Reading the code, that is not what it does. What is in main.py is a webcam capture loop that runs MediaPipe Holistic on each frame, draws the face mesh, hand, and pose landmarks onto a background image, saves each frame, and stitches them into a GIF. It is a pose extractor and a GIF exporter. It has no text input. It has no generation model. It is closer to a data collection tool for the first repo than to a text-to-sign pipeline.
I could dress that up. I could say I was scoping the harder problem into a data pipeline first. That would be revisionist. The truth is that generation is much harder than recognition, and I hit the wall the moment I tried. To generate a sign, you need a model of what a well-formed sign looks like in the pose space, and a way to condition it on a target word or morpheme. The good academic work uses transformers over pose sequences with attention on the input gloss, or diffusion in a learned pose latent. I had a laptop, a webcam, MediaPipe, and no dataset. So I built the piece I could build. I called it "generation" because I hoped it would grow into that, and then it did not.
If you clone that repo today, you get a MediaPipe holistic recorder. That is worth being direct about.
Repo three: audio to sign, with a shortcut
The third repo, Audio-to-Sign in August 2023, is a Streamlit app. The user speaks into the mic, the app runs Google's speech recogniser, and then it looks up each letter of the recognised text in an Alphabets/ folder and displays the fingerspelling image for that letter, in order.
The obvious question is why go audio to text to fingerspelling, rather than trying to go from audio directly to sign. Two reasons, one honest, one interesting. The honest reason is that off-the-shelf speech recognition was solved and free, and off-the-shelf sign generation did not exist. Pushing the modality bridge as far as the text layer let me use good components for one half and a lookup table for the other. The interesting reason is that any real audio-to-sign system has to route through some kind of intermediate meaning. Speech to gloss to sign is closer to how a human interpreter actually works than speech directly to hand motion. My repo does not have a gloss layer, but the shape of speech, then symbolic intermediate, then visual output, is the right general shape.
The shortcut, of course, is that fingerspelling every word in a spoken sentence is not how any deaf signer would want to be communicated with. Imagine reading English if every message was spelled letter by letter, with no words. That is what the app produces. It is a working demo of a pipeline whose output nobody in the target community would sit through.
What the trilogy actually taught me
Reading these three repos back to back is uncomfortable in a useful way. The technical curve is real. In September 2022 I was training a sequence model on hand keypoints. By August 2023 I was wiring three services (speech recognition, a Streamlit UI, an image lookup) into one app. The engineering got broader. The accessibility got worse.
Here is the thing I did not understand at 21 and would tell any student picking up this problem now. Accuracy on a benchmark is not the same as usable in a conversation. My LSTM's 0.7 categorical accuracy on three fingerspelled letters, in my lighting, on my hand, is a number. It does not tell me whether a deaf person could use this to order coffee. My audio-to-fingerspelling app returns something for every sentence you speak. That does not mean it communicates. The last mile of every accessibility system is the human on the other end deciding whether the output is worth their time. If it is not, the accuracy is irrelevant.
The version of this project I would build now would start from the other end. I would talk to actual signers, watch what breaks in existing tools, and only then decide whether the useful thing is a recogniser, a generator, or a completely different piece of tooling (real-time captioning, better video call layouts, gloss-aware speech recognition). The three repos, taken together, are what happens when a student picks the hardest looking version of a problem and works backward toward what they can build in a weekend. The commits are honest. The framing was not. I am leaving them public (except the private one) because they are useful evidence of a specific mistake, and because a portfolio that only shows the good runs is not a portfolio, it is a highlight reel.
Adjacent to the three, there is a fourth repo called Video-edit-Audio-sync from the same stretch (August 2023), a Streamlit app that reads tempo from an uploaded mp3 with librosa.beat.beat_track and applies a smooth per-frame shake to an uploaded video whose intensity decays over the clip. It never made it into any of the three pipelines, but the beat-tracking-to-per-frame-timing wiring is what I would reach for now if I ever came back to a gloss-timed sign generator.
If the three repos have a single lesson, it is that in accessibility ML, the model is the easy part.