This is a submission for the Presage SmartSpectra SDK Challenge
What I Built
Traditional Augmentative and Alternative Communication (AAC) systems for people with severe motor and speech impairments (such as ALS, high-level quadriplegia, or locked-in syndrome) force an impossible trade-off: spend $3,000+ on specialized infrared eye-tracking hardware, or endure exhausting dwell-time interfaces where staring at a button for 1,000 ms just to register a single click leads to rapid visual fatigue and constant accidental selections (the classic "Midas Touch" problem).
Worse, existing gaze communicators suffer from two real-world bedside failures that nobody talks about:
- Attention Theft: The screen holds the user's eyes hostage. If they glance up to make eye contact with a loved one or try to vocalize, the cursor misfires and types gibberish.
- One-Sided Visibility: Caregivers and family members standing across the room can't read a laptop screen facing the patient.
NeuroScribe Studio turns any standard consumer webcam into a responsive, spatial assistive communication console by decoupling targeting from activation, introducing conversational standby gating, and syncing messages in real time to a Caregiver Partner HUD (/partner).
Core Capabilities
- 3D Spatial Gaze & Head-Pose Targeting: High-contrast, multi-level hierarchical communication boards (Physical Needs, Comfort & Position, People & Care, Quick Dialogue, Expressions, Emergency) driven by smoothed iris vectors and head pitch/yaw.
-
Temporal Blink Discrimination: A client-side state machine separates natural involuntary blinks (
< 250 ms, ignored) from deliberate selection blinks (350–850 ms, confirms card) and extended closures (> 1,200 ms, undo/backspace). -
Attentive Standby Lockout: Automatically freezes card inputs the instant
face.talkingis detected or the user tilts their head up (pitch > 14°) to make natural eye contact with a caregiver. -
Interactive 5-Second Calibration: Dynamically normalizes iris bounding thresholds (
minX, maxX, minY, maxY) and blink timing windows to adapt to different seating angles, wheelchair heights, and eyelid control speeds. -
Cross-Device Caregiver Mirror (
/partner): Real-time WebSocket sync broadcasts composed phrases, spoken audio chimes, and haptic vibration alerts (navigator.vibrate) to a caregiver's smartphone across the room—complete with two-way acknowledgment back to the patient. -
Human-in-the-Loop Speech Synthesis: Messages are never spoken automatically; users review their constructed sentence and confirm
[Speak Aloud]to trigger the browser's native Web Speech API.
Demo
What Happens in the Demo:
- Consent & Privacy Gate: Explicit pre-capture authorization explaining local in-memory processing and zero telemetry.
- Hands-Free Spatial Navigation: Navigating the 3D spatial wave-cards via eye gaze and selecting phrases with deliberate blinks—with hands completely off the keyboard.
- Attentive Standby Lockout: Looking up and speaking freezes the grid so natural eye contact with a caregiver never causes accidental selections.
-
Caregiver Mirror Sync (
/partner): Confirming Speak Aloud broadcasts the message in real time to a second device with audio and haptic feedback.
Code
- Public GitHub Repository: https://github.com/Akulbanxal/Neuroscribe
How It Responds to Human Signals
Rather than treating SmartSpectra as a passive telemetry display, NeuroScribe Studio orchestrates four live Presage human signals in real time to power a hands-free, zero-fatigue assistive communication loop:
-
Iris Vectors & 478 Facial Landmarks (
faceMetrics) → Spatial Card Targeting & 3D Parallax:- Normalized iris coordinates (
[0, 1]) and head pose (pitchandyaw) are streamed from the local SmartSpectra Node SDK daemon (@smartspectra/node-sdk) at 30 FPS over WebSockets (ws://localhost:8080). - To eliminate raw webcam jitter, the frontend applies an Exponential Moving Average (EMA) filter (α = 0.28–0.35) and maps the smoothed gaze vector onto calibrated 3×2 spatial wave-cards with deadzones, dynamically tilting the 3D glass deck in response to head orientation.
- Normalized iris coordinates (
-
Binary Blink Detection (
face.blinking) → Temporal Selection & Undo State Machine:- Instead of exhausting dwell-time timers (staring at a button for 1,000+ ms), NeuroScribe measures the exact duration of continuous
face.blinking.detectedwindows on the client side:-
Natural involuntary blinks (
< 250 ms): Ignored completely to prevent accidental clicks. -
Deliberate selection blinks (
350–850 ms): Fills a live neon progress bar on the targeted card and appends the phrase to the sentence builder. -
Extended eye closure (
> 1,200 ms): Triggers an instant Undo / Backspace action.
-
Natural involuntary blinks (
- Instead of exhausting dwell-time timers (staring at a button for 1,000+ ms), NeuroScribe measures the exact duration of continuous
-
Speech Articulation Detection (
face.talking) & Head Pitch → Attentive Standby Lockout:- The moment
face.talking.detectedflips totrue—or when the user lifts their chin (pitch > 14°) to make natural eye contact with a bedside caregiver—the UI immediately enters Attentive Standby, freezing all card selections and dimming the grid so real-world conversation never triggers false inputs.
- The moment
-
Respiratory Waveform (
breathing.upperTrace) & Facial Expressions (faceMetrics.expressions) → Live HUD & Ambient Feedback:- Real-time upper-chest respiratory excursions drive the rhythmic breathing sparkline in the system status bar, while dominant facial expression probabilities subtly adapt the ambient spatial lighting.
SDK Integration Architecture
-
Backend Bridge (
server.mjs): InitializesSmartSpectraSDKlocally via C++ FFI withrequestedMetrics: [...faceMetrics, ...breathingMetrics]andenableTelemetry: false, normalizing all frame timestamps to epoch microseconds (Date.now() * 1000) to guarantee continuous, stall-free streaming. -
Frontend Consumer (
useBiometrics.ts): Ingests the 30 FPS WebSocket JSON stream, applies EMA coordinate smoothing, runs the temporal blink discriminator, and synchronizes confirmed phrases to both the Web Speech API (speechSynthesis) and the cross-device Caregiver Mirror (/partner).
Prize Categories
I am entering NeuroScribe Studio for Best Thoughtful Real-World Use (Primary) as well as Best Overall Human-Sensing Experience.
Best Thoughtful Real-World Use
- The Real-World Need: Millions of individuals living with severe motor and speech impairments—such as ALS, high-level spinal cord injuries, cerebral palsy, or brainstem stroke—lose the ability to speak or use touchscreens. Existing Augmentative and Alternative Communication (AAC) setups either cost $3,000+ for dedicated infrared hardware or rely on exhausting dwell-time timers (staring at a button for 1,000+ ms to click) that cause rapid ocular fatigue and constant accidental inputs.
- Who It’s For: Non-speaking or motor-impaired individuals who need a reliable, zero-cost way to express urgent needs, comfort adjustments, and everyday thoughts using only a standard laptop webcam—alongside their bedside caregivers and family members who need immediate visibility across the room.
-
How It Was Designed for Real-World Usability:
-
Zero-Fatigue Temporal Blink Activation: Decouples eye-gaze targeting from intentional blink confirmation (
350–850 ms), ignoring involuntary natural blinks (< 250 ms) and mapping extended closures (> 1,200 ms) to Undo. -
Attentive Standby Lockout: Automatically freezes card inputs when
face.talkingis detected or when the user tilts their head up (pitch > 14°) to make natural eye contact with a loved one—eliminating accidental selections during real-world conversation. - 5-Second Adaptive Calibration & High-Contrast Spatial Cards: Large, unmistakable 3×2 wave-cards with customizable blink thresholds and seating calibration accommodate varying wheelchair angles and eyelid control speeds.
-
Cross-Device Caregiver Mirror (
/partner): Broadcasts composed phrases, spoken audio chimes, and haptic vibrations (navigator.vibrate) to a caregiver's smartphone in real time, complete with two-way acknowledgment back to the patient.
-
Zero-Fatigue Temporal Blink Activation: Decouples eye-gaze targeting from intentional blink confirmation (
Consent and Data Handling
NeuroScribe Studio was architected from the ground up with strict privacy-first guardrails and full compliance with the challenge rules:
- Explicit Pre-Capture Consent: Camera access and WebSocket biometric streaming remain completely disabled at launch until the user explicitly reviews and accepts the on-screen Camera Consent & Privacy Notice Modal (with a one-click option to revoke/pause capture at any time).
-
100% Local In-Memory Processing (Zero Storage / Zero Sharing): Video frames are captured locally by the Node.js daemon (
@smartspectra/node-sdk), processed strictly in volatile RAM, and immediately discarded. Zero video frames, facial images, landmark recordings, or biometric logs are ever saved to disk, persisted in databases, or transmitted to external servers. - Retention & Deletion: Because all processing occurs ephemerally in volatile memory with zero disk persistence, data retention is 0 seconds—nothing is stored, so no residual user data ever remains after closing the session.
-
SDK Telemetry Disabled: The SmartSpectra SDK instance is explicitly initialized with
enableTelemetry: false, ensuring no diagnostic session telemetry or usage metrics leave the local machine. -
No Third-Party Cloud AI Services: All gaze smoothing, blink classification, sentence assembly, and text-to-speech synthesis (
window.speechSynthesis) run 100% locally on the user's device without calling external AI APIs. - Non-Medical & Non-Diagnostic Disclosure: NeuroScribe Studio is strictly an assistive human-computer interaction (HCI) communication interface. It makes no medical, diagnostic, or treatment claims, performs no clinical decision-making, and does not enable covert or non-consensual monitoring.
Agent Session
Built solo using React 19, TypeScript, Vite, Tailwind CSS, and @smartspectra/node-sdk, with AI-assisted architectural iteration and UI prototyping.
Top comments (9)
The blink windows are what make this feel real: under 250 ms ignored, 350 to 850 ms to select, over 1,200 ms to undo. The standby lockout when someone looks up at a caregiver is the kind of detail you only add after watching a person use one of these. Paul's question about blink timing drifting as the user tires is the one I would want answered next.
The bigger idea I take from it is that a lot of costly or locked-down health hardware is really a sensor plus software, and the software is where people get stuck. At Ascendra Ventures we hit a much smaller version of this with Actofit body-composition scales. The hardware still worked, but the official app stopped returning the full readout. A scale like that only sends two raw numbers over Bluetooth, weight and impedance, and everything else is calculated. So we rebuilt it as a web app that reads those two numbers through Web Bluetooth in the browser and recomputes the derived metrics from the user's profile, with no install and no cloud in the middle.
Your local, in-memory processing gives the same benefit for a far more sensitive use. Do you plan to let the calibration be saved on the device, so a caregiver does not need to repeat it every session?
Love that example with the Actofit scale—it's the exact same problem. The camera hardware is already more than capable, it's just trapped behind expensive, locked-down software.
For the challenge submission I kept everything strictly in RAM so nothing touches disk by default. But you're completely right about daily use—forcing a caregiver to recalibrate every single time a tab reloads gets annoying fast.
The plan is to add an opt-in "Save Profile" toggle in localStorage that only stores the 4 gaze boundary numbers and the blink millisecond range, with a one-click reset button in the top bar. No camera frames or biometric logs ever get saved, so it stays 100% private while letting a caregiver set the seating angle once and forget it.
The 350–850ms blink window is the detail that makes this believable — that's a real discrimination problem, not a demo trick. And the standby lockout when someone tilts their head up to talk to a caregiver is the kind of thing you only build if you've actually watched someone use one of these. How does the calibration hold up when the user gets tired and their blink timing drifts over a session?
Paul —ocular fatigue definitely slows down intentional blinks over a long session. Right now, two things prevent lockouts:
Live Visual Fill Bar: The targeted card displays a real-time charging bar during the blink hold, so the user sees the exact moment to release rather than guessing.
On-the-Fly Tuner: The Calibrate tab includes a live Blink Threshold slider to widen the window (e.g., 450–1,050ms) anytime without restarting.
Next up on my roadmap is an auto-adapting EMA tracker on confirmed blinks so the upper threshold automatically widens as the user's blink duration drifts!
Really clean approach to solving the Midas Touch problem without expensive IR hardware. Using face.talking and head pitch to trigger an automatic standby lockout is such a thoughtful detail—most gaze interfaces completely miss how awkward it is when someone tries to make eye contact with a person in the room. Great execution!
Replacing $3k eye-trackers with a standard webcam + SmartSpectra is seriously impressive. The auto-standby during conversation and the /partner sync make this stand out as a genuinely usable AAC tool. Awesome build!
Love the spatial wave-card UI and the 350–850ms blink discrimination window—way less exhausting than traditional 1-second dwell timers. Having a built-in 30 FPS fallback stream when no camera is attached is also a super smart touch for testing
Really impressive work! Loved the concept and execution.
THE THING I LOVED THE MOSTIS THE FACT THAT ALL HAS BEEN DONE THROUGH RAW ENERGY AND PASSION ABOUT SOLVING PROBLEMS