Step 1
Track the signer.
MediaPipe Pose and Hands run in the browser and track 75 body and hand landmarks in every frame of the webcam clip.
Accessibility · Computer vision
Sign in ASL. Read English. Hear it spoken.
SignSpeak is a proof of concept I built for DECA's Entrepreneurship Innovation Plan. Someone signs on camera, a dedicated model reads the signs, and the browser speaks the English translation out loud.
A 200-class BiLSTM trained on ASL Citizen keypoints, then fine-tuned with reinforcement learning.
Capture, classification and speech are separate steps, so each one can be shown and explained on its own.
Step 1
MediaPipe Pose and Hands run in the browser and track 75 body and hand landmarks in every frame of the webcam clip.
Step 2
The landmarks are normalized, resampled and passed to a BiLSTM on the server through ONNX Runtime. It only answers when the confidence checks pass.
Step 3
Gemini turns the recognized signs into a natural English sentence, and the Web Speech API reads it aloud with a replay button.
Sign one word, or several with a short pause after each. The result shows which path produced it: the dedicated model or Gemini video.
A prediction is accepted only if the top score is at least 55%, it leads the runner-up by 0.15, and entropy is low. Otherwise the clip goes to Gemini video instead of showing a bad guess.
Clips up to about 30 seconds are split wherever the hands pause for 400 ms. Each segment goes through the same checks, and the sentence is only built if more than half of them pass.
A linear layer, a two-layer bidirectional LSTM and attention pooling over 450 features per frame. Reinforcement learning lifted test accuracy from about 87.3% to 90.3%.
It reads isolated signs, not fluent conversation, and accuracy depends on lighting and framing. It is a feasibility demo, not a substitute for a human interpreter.