DEV Community

Cover image for I Replaced $3,000 Eye-Trackers with a Laptop Webcam: Building a Hands-Free Spatial Communicator with SmartSpectra
Akul Bansal
Akul Bansal

Posted on AI-assisted

I Replaced $3,000 Eye-Trackers with a Laptop Webcam: Building a Hands-Free Spatial Communicator with SmartSpectra

Presage SmartSpectra SDK Challenge Submission

This is a submission for the Presage SmartSpectra SDK Challenge

What I Built

Traditional Augmentative and Alternative Communication (AAC) systems for people with severe motor and speech impairments (such as ALS, high-level quadriplegia, or locked-in syndrome) force an impossible trade-off: spend $3,000+ on specialized infrared eye-tracking hardware, or endure exhausting dwell-time interfaces where staring at a button for 1,000 ms just to register a single click leads to rapid visual fatigue and constant accidental selections (the classic "Midas Touch" problem).

Worse, existing gaze communicators suffer from two real-world bedside failures that nobody talks about:

  1. Attention Theft: The screen holds the user's eyes hostage. If they glance up to make eye contact with a loved one or try to vocalize, the cursor misfires and types gibberish.
  2. One-Sided Visibility: Caregivers and family members standing across the room can't read a laptop screen facing the patient.

NeuroScribe Studio turns any standard consumer webcam into a responsive, spatial assistive communication console by decoupling targeting from activation, introducing conversational standby gating, and syncing messages in real time to a Caregiver Partner HUD (/partner).

Core Capabilities

  • 3D Spatial Gaze & Head-Pose Targeting: High-contrast, multi-level hierarchical communication boards (Physical Needs, Comfort & Position, People & Care, Quick Dialogue, Expressions, Emergency) driven by smoothed iris vectors and head pitch/yaw.
  • Temporal Blink Discrimination: A client-side state machine separates natural involuntary blinks (< 250 ms, ignored) from deliberate selection blinks (350–850 ms, confirms card) and extended closures (> 1,200 ms, undo/backspace).
  • Attentive Standby Lockout: Automatically freezes card inputs the instant face.talking is detected or the user tilts their head up (pitch > 14°) to make natural eye contact with a caregiver.
  • Interactive 5-Second Calibration: Dynamically normalizes iris bounding thresholds (minX, maxX, minY, maxY) and blink timing windows to adapt to different seating angles, wheelchair heights, and eyelid control speeds.
  • Cross-Device Caregiver Mirror (/partner): Real-time WebSocket sync broadcasts composed phrases, spoken audio chimes, and haptic vibration alerts (navigator.vibrate) to a caregiver's smartphone across the room—complete with two-way acknowledgment back to the patient.
  • Human-in-the-Loop Speech Synthesis: Messages are never spoken automatically; users review their constructed sentence and confirm [Speak Aloud] to trigger the browser's native Web Speech API.

Demo

What Happens in the Demo:

  1. Consent & Privacy Gate: Explicit pre-capture authorization explaining local in-memory processing and zero telemetry.
  2. Hands-Free Spatial Navigation: Navigating the 3D spatial wave-cards via eye gaze and selecting phrases with deliberate blinks—with hands completely off the keyboard.
  3. Attentive Standby Lockout: Looking up and speaking freezes the grid so natural eye contact with a caregiver never causes accidental selections.
  4. Caregiver Mirror Sync (/partner): Confirming Speak Aloud broadcasts the message in real time to a second device with audio and haptic feedback.

Code

How It Responds to Human Signals

Rather than treating SmartSpectra as a passive telemetry display, NeuroScribe Studio orchestrates four live Presage human signals in real time to power a hands-free, zero-fatigue assistive communication loop:

  1. Iris Vectors & 478 Facial Landmarks (faceMetrics) → Spatial Card Targeting & 3D Parallax:

    • Normalized iris coordinates ([0, 1]) and head pose (pitch and yaw) are streamed from the local SmartSpectra Node SDK daemon (@smartspectra/node-sdk) at 30 FPS over WebSockets (ws://localhost:8080).
    • To eliminate raw webcam jitter, the frontend applies an Exponential Moving Average (EMA) filter (α = 0.28–0.35) and maps the smoothed gaze vector onto calibrated 3×2 spatial wave-cards with deadzones, dynamically tilting the 3D glass deck in response to head orientation.
  2. Binary Blink Detection (face.blinking) → Temporal Selection & Undo State Machine:

    • Instead of exhausting dwell-time timers (staring at a button for 1,000+ ms), NeuroScribe measures the exact duration of continuous face.blinking.detected windows on the client side:
      • Natural involuntary blinks (< 250 ms): Ignored completely to prevent accidental clicks.
      • Deliberate selection blinks (350–850 ms): Fills a live neon progress bar on the targeted card and appends the phrase to the sentence builder.
      • Extended eye closure (> 1,200 ms): Triggers an instant Undo / Backspace action.
  3. Speech Articulation Detection (face.talking) & Head Pitch → Attentive Standby Lockout:

    • The moment face.talking.detected flips to true—or when the user lifts their chin (pitch > 14°) to make natural eye contact with a bedside caregiver—the UI immediately enters Attentive Standby, freezing all card selections and dimming the grid so real-world conversation never triggers false inputs.
  4. Respiratory Waveform (breathing.upperTrace) & Facial Expressions (faceMetrics.expressions) → Live HUD & Ambient Feedback:

    • Real-time upper-chest respiratory excursions drive the rhythmic breathing sparkline in the system status bar, while dominant facial expression probabilities subtly adapt the ambient spatial lighting.

SDK Integration Architecture

  • Backend Bridge (server.mjs): Initializes SmartSpectraSDK locally via C++ FFI with requestedMetrics: [...faceMetrics, ...breathingMetrics] and enableTelemetry: false, normalizing all frame timestamps to epoch microseconds (Date.now() * 1000) to guarantee continuous, stall-free streaming.
  • Frontend Consumer (useBiometrics.ts): Ingests the 30 FPS WebSocket JSON stream, applies EMA coordinate smoothing, runs the temporal blink discriminator, and synchronizes confirmed phrases to both the Web Speech API (speechSynthesis) and the cross-device Caregiver Mirror (/partner).

Prize Categories

I am entering NeuroScribe Studio for Best Thoughtful Real-World Use (Primary) as well as Best Overall Human-Sensing Experience.

Best Thoughtful Real-World Use

  • The Real-World Need: Millions of individuals living with severe motor and speech impairments—such as ALS, high-level spinal cord injuries, cerebral palsy, or brainstem stroke—lose the ability to speak or use touchscreens. Existing Augmentative and Alternative Communication (AAC) setups either cost $3,000+ for dedicated infrared hardware or rely on exhausting dwell-time timers (staring at a button for 1,000+ ms to click) that cause rapid ocular fatigue and constant accidental inputs.
  • Who It’s For: Non-speaking or motor-impaired individuals who need a reliable, zero-cost way to express urgent needs, comfort adjustments, and everyday thoughts using only a standard laptop webcam—alongside their bedside caregivers and family members who need immediate visibility across the room.
  • How It Was Designed for Real-World Usability:
    • Zero-Fatigue Temporal Blink Activation: Decouples eye-gaze targeting from intentional blink confirmation (350–850 ms), ignoring involuntary natural blinks (< 250 ms) and mapping extended closures (> 1,200 ms) to Undo.
    • Attentive Standby Lockout: Automatically freezes card inputs when face.talking is detected or when the user tilts their head up (pitch > 14°) to make natural eye contact with a loved one—eliminating accidental selections during real-world conversation.
    • 5-Second Adaptive Calibration & High-Contrast Spatial Cards: Large, unmistakable 3×2 wave-cards with customizable blink thresholds and seating calibration accommodate varying wheelchair angles and eyelid control speeds.
    • Cross-Device Caregiver Mirror (/partner): Broadcasts composed phrases, spoken audio chimes, and haptic vibrations (navigator.vibrate) to a caregiver's smartphone in real time, complete with two-way acknowledgment back to the patient.

Consent and Data Handling

NeuroScribe Studio was architected from the ground up with strict privacy-first guardrails and full compliance with the challenge rules:

  • Explicit Pre-Capture Consent: Camera access and WebSocket biometric streaming remain completely disabled at launch until the user explicitly reviews and accepts the on-screen Camera Consent & Privacy Notice Modal (with a one-click option to revoke/pause capture at any time).
  • 100% Local In-Memory Processing (Zero Storage / Zero Sharing): Video frames are captured locally by the Node.js daemon (@smartspectra/node-sdk), processed strictly in volatile RAM, and immediately discarded. Zero video frames, facial images, landmark recordings, or biometric logs are ever saved to disk, persisted in databases, or transmitted to external servers.
  • Retention & Deletion: Because all processing occurs ephemerally in volatile memory with zero disk persistence, data retention is 0 seconds—nothing is stored, so no residual user data ever remains after closing the session.
  • SDK Telemetry Disabled: The SmartSpectra SDK instance is explicitly initialized with enableTelemetry: false, ensuring no diagnostic session telemetry or usage metrics leave the local machine.
  • No Third-Party Cloud AI Services: All gaze smoothing, blink classification, sentence assembly, and text-to-speech synthesis (window.speechSynthesis) run 100% locally on the user's device without calling external AI APIs.
  • Non-Medical & Non-Diagnostic Disclosure: NeuroScribe Studio is strictly an assistive human-computer interaction (HCI) communication interface. It makes no medical, diagnostic, or treatment claims, performs no clinical decision-making, and does not enable covert or non-consensual monitoring.

Agent Session

Built solo using React 19, TypeScript, Vite, Tailwind CSS, and @smartspectra/node-sdk, with AI-assisted architectural iteration and UI prototyping.

Top comments (9)

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

The blink windows are what make this feel real: under 250 ms ignored, 350 to 850 ms to select, over 1,200 ms to undo. The standby lockout when someone looks up at a caregiver is the kind of detail you only add after watching a person use one of these. Paul's question about blink timing drifting as the user tires is the one I would want answered next.

The bigger idea I take from it is that a lot of costly or locked-down health hardware is really a sensor plus software, and the software is where people get stuck. At Ascendra Ventures we hit a much smaller version of this with Actofit body-composition scales. The hardware still worked, but the official app stopped returning the full readout. A scale like that only sends two raw numbers over Bluetooth, weight and impedance, and everything else is calculated. So we rebuilt it as a web app that reads those two numbers through Web Bluetooth in the browser and recomputes the derived metrics from the user's profile, with no install and no cloud in the middle.

Your local, in-memory processing gives the same benefit for a far more sensitive use. Do you plan to let the calibration be saved on the device, so a caregiver does not need to repeat it every session?

Collapse
 
akul_9512534b6f460d338911 profile image
Akul Bansal •

Love that example with the Actofit scale—it's the exact same problem. The camera hardware is already more than capable, it's just trapped behind expensive, locked-down software.

For the challenge submission I kept everything strictly in RAM so nothing touches disk by default. But you're completely right about daily use—forcing a caregiver to recalibrate every single time a tab reloads gets annoying fast.

The plan is to add an opt-in "Save Profile" toggle in localStorage that only stores the 4 gaze boundary numbers and the blink millisecond range, with a one-click reset button in the top bar. No camera frames or biometric logs ever get saved, so it stays 100% private while letting a caregiver set the seating angle once and forget it.

Collapse
 
paullovelace profile image
Paul Lovelace •

The 350–850ms blink window is the detail that makes this believable — that's a real discrimination problem, not a demo trick. And the standby lockout when someone tilts their head up to talk to a caregiver is the kind of thing you only build if you've actually watched someone use one of these. How does the calibration hold up when the user gets tired and their blink timing drifts over a session?

Collapse
 
akul_9512534b6f460d338911 profile image
Akul Bansal •

Paul —ocular fatigue definitely slows down intentional blinks over a long session. Right now, two things prevent lockouts:

Live Visual Fill Bar: The targeted card displays a real-time charging bar during the blink hold, so the user sees the exact moment to release rather than guessing.
On-the-Fly Tuner: The Calibrate tab includes a live Blink Threshold slider to widen the window (e.g., 450–1,050ms) anytime without restarting.
Next up on my roadmap is an auto-adapting EMA tracker on confirmed blinks so the upper threshold automatically widens as the user's blink duration drifts!

Collapse
 
sahil_lulani_314ec1d8d0b8 profile image
Sahil Lulani •

Really clean approach to solving the Midas Touch problem without expensive IR hardware. Using face.talking and head pitch to trigger an automatic standby lockout is such a thoughtful detail—most gaze interfaces completely miss how awkward it is when someone tries to make eye contact with a person in the room. Great execution!

Collapse
 
andrej_9d047df9f75 profile image
anderj_koshnov •

Replacing $3k eye-trackers with a standard webcam + SmartSpectra is seriously impressive. The auto-standby during conversation and the /partner sync make this stand out as a genuinely usable AAC tool. Awesome build!

Collapse
 
sarwan_kumar_ef20e773c7d7 profile image
SARWAN Kumar •

Love the spatial wave-card UI and the 350–850ms blink discrimination window—way less exhausting than traditional 1-second dwell timers. Having a built-in 30 FPS fallback stream when no camera is attached is also a super smart touch for testing

Collapse
 
bhavya_bansal_513001ba9f0 profile image
Bhavya Bansal •

Really impressive work! Loved the concept and execution.

Collapse
 
mohit_sangwan_e8e33022c16 profile image
Mohit Sangwan •

THE THING I LOVED THE MOSTIS THE FACT THAT ALL HAS BEEN DONE THROUGH RAW ENERGY AND PASSION ABOUT SOLVING PROBLEMS