What I Built
Prapatthi AI is an accessibility-first, camera-driven human-computer interaction system designed for individuals with severe motor impairments, limb differences, or paralysis who cannot use traditional keyboards, mice, or touchscreens.
Standard assistive tools often rely on expensive eye-tracking rigs, sip-and-puff hardware, or fragile single-gesture triggers that cause rapid user fatigue. Prapatthi AI turns a standard webcam or mobile front camera into an adaptive, hands-free control interface. It allows users to navigate desktop and mobile operating systems, trigger clicks, scroll, switch contexts, and pace their interactions automatically based on their real-time physiological and expressive state.
Code
Explore the source code, architecture, and local setup instructions in the public repository:github.com/Sai-Emani25/Prapatthi-AI
How It Responds to Human Signals
Prapatthi AI integrates the Presage SmartSpectra SDK to capture continuous, non-contact physiological and facial telemetry directly from the camera feed and translate those signals into both explicit navigation commands and implicit UI adaptation:
Facial Expressions & Micro-Movements (Intentional Control): Intentional facial cues—such as sustained brow raises, deliberate blinks, and subtle head/jaw expression shifts—are mapped to deterministic OS actions (cursor selection, left/right click, scrolling, and action confirmation) without requiring physical touch.
Pulse & Heart Rate Variability (Cognitive Load & Fatigue Detection): Continuous rPPG pulse and HRV trends monitor user exertion and frustration in real time. When elevated stress or interaction fatigue is detected during fine-grained navigation tasks, Prapatthi AI automatically enlarges target hitboxes, slows cursor dwell timers, and reduces multi-step menu depth so the user never has to fight the interface.
Breathing Rate (Rhythmic Pacing & Calm Confirmation): Breathing cadence stabilizes dwell-to-click thresholds and provides a low-effort, low-strain confirmation mechanism for users who experience muscle spasticity or tremors.
SDK Integration Pipeline
Frame Capture & Consent Gate: Local camera frames are ingested only after an explicit opt-in prompt is accepted.
SmartSpectra Signal Extraction: The SmartSpectra SDK processes live video frames to stream structured metrics (pulse, breathing_rate, hrv, and facial expression vectors) at low latency.
Adaptive Action Engine: A lightweight state machine debounces intentional expression triggers (preventing accidental clicks from involuntary movements) while feeding physiological stress/fatigue baselines into a dynamic UI scaling and dwell-time controller.
OS Bridge: Verified commands are dispatched to native desktop and mobile accessibility APIs to execute clicks, scrolls, and keyboard shortcuts seamlessly.
Prize Categories
Best Thoughtful Real-World Use
The Real-World Need: Millions of people living with spinal cord injuries, ALS, cerebral palsy, or limb loss face daily friction using modern laptops and smartphones built around fine motor dexterity—and commercial adaptive hardware is often cost-prohibitive or tiring to operate over long sessions.
Who It Is For: Individuals with motor disabilities who need a zero-cost, hardware-free way to navigate digital workspaces and mobile devices using only a built-in camera.
Thoughtful Usability Design: Rather than treating human signals purely as rigid binary switches, Prapatthi AI uses physiological signals (pulse, HRV, breathing) to make the interface empathize with the user's physical effort—expanding buttons and relaxing timing windows the moment fatigue sets in, while keeping every control customizable to the user's exact range of motion.
Consent and Data Handling
Explicit Camera Consent: Camera access is disabled by default. On launch, Prapatthi AI presents a clear, plain-language consent screen explaining what signals are read and requires an explicit user confirmation before initializing the video stream. A persistent on-screen indicator shows whenever the camera is active, alongside a single-trigger pause/kill switch.
Local-First Processing & Zero Video Storage: Raw camera frames are processed transiently in memory solely to extract real-time control signals via the Presage SmartSpectra SDK. Video frames and biometric recordings are never saved to disk, logged, or transmitted to external servers.
No Third-Party Sharing or Profiling: Derived signal states exist only for the duration of the active navigation session and are immediately discarded when the session ends or the camera is paused. No user profiles, biometric identifiers, or telemetry are retained or shared with third parties.
Non-Medical Scope: Prapatthi AI is strictly an assistive human-computer interaction and accessibility navigation tool. It does not diagnose, monitor, treat, or provide medical insights for any health condition, and all processing is strictly user-initiated and consensual.
Top comments (0)