Accessibility · Computer vision

SignSpeak

Sign in ASL. Read English. Hear it spoken.

SignSpeak landing page explaining the webcam to spoken English loop.

SignSpeak is a proof of concept I built for DECA's Entrepreneurship Innovation Plan. Someone signs on camera, a dedicated model reads the signs, and the browser speaks the English translation out loud.

The numbers.

A 200-class BiLSTM trained on ASL Citizen keypoints, then fine-tuned with reinforcement learning.

ASL signs in the vocabulary
200
Top-1 test accuracy
90.3%
Top-5 test accuracy
98.1%
Longest clip, split at pauses
30s

How it works.

Capture, classification and speech are separate steps, so each one can be shown and explained on its own.

Step 1

Track the signer.

MediaPipe Pose and Hands run in the browser and track 75 body and hand landmarks in every frame of the webcam clip.

Step 2

Read the sign.

The landmarks are normalized, resampled and passed to a BiLSTM on the server through ONNX Runtime. It only answers when the confidence checks pass.

Step 3

Say it in English.

Gemini turns the recognized signs into a natural English sentence, and the Web Speech API reads it aloud with a replay button.

The camera demo.

Sign one word, or several with a short pause after each. The result shows which path produced it: the dedicated model or Gemini video.

SignSpeak camera studio with a camera panel, Start signing button, and an English translation panel with Replay voice.

Under the hood.

Confidence gates.

A prediction is accepted only if the top score is at least 55%, it leads the runner-up by 0.15, and entropy is low. Otherwise the clip goes to Gemini video instead of showing a bad guess.

Several signs in a row.

Clips up to about 30 seconds are split wherever the hands pause for 400 ms. Each segment goes through the same checks, and the sentence is only built if more than half of them pass.

The model.

A linear layer, a two-layer bidirectional LSTM and attention pooling over 450 features per frame. Reinforcement learning lifted test accuracy from about 87.3% to 90.3%.

Honest limits.

It reads isolated signs, not fluent conversation, and accuracy depends on lighting and framing. It is a feasibility demo, not a substitute for a human interpreter.

Tech specs.

Frontend
Next.js 16, React 19, TypeScript, Tailwind CSS
Vision
MediaPipe Tasks (Pose and Hands) in the browser
Model
200-class BiLSTM with attention pooling, served with ONNX Runtime on Node
Training data
ASL Citizen keypoints, supervised training then reinforcement learning
Language
Google Gemini for gloss-to-English and the full-video fallback
Speech
Web Speech API
Hosting
Vercel
Next projectLocus
All projects