Projects

Commute Booster is a mobile navigation application designed to support people with blindness and low vision during subway travel, particularly within complex transit stations. It combines GTFS-based trip planning with real-time sign recognition using computer vision, enabling users to locate and confirm wayfinding signage inside stations without additional infrastructure.

• Full trip planning plus real-time station guidance
• Phone camera sign recognition using OCR
• GTFS-derived sign list aligned to the intended route
• Validated in NYC stations with strong accuracy

Point-to-Tell 2 is a selective, gesture-controlled assistive system that helps people who are blind or have low vision request only the information they want, reducing extraneous feedback and potential cognitive overload. Using a monocular camera with AI pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system infers a user's 3D pointing direction and announces the names and distances of pointed-at objects.

• Pointing-based selection to control what the system describes
• 3D ray-casting links object identity with spatial position
• AI stack: depth estimation, hand pose tracking, detection/segmentation
• Private, hands-free interaction designed for real-world use

ShopScout is a multimodal assistive system that enables people with blindness and low vision to shop independently. By combining object detection, vision-language models, and spatial audio feedback, the system guides users from shopping list creation to successful item retrieval without requiring assistance.

• End-to-end workflow from list creation to item retrieval
• Real-time object detection for product identification
• Vision-language models for contextual understanding
• Spatial audio feedback for intuitive navigation

Virtual Whiskers is a haptics-based electronic travel aid designed to support obstacle negotiation for people with blindness and low vision. Using modular vibration units and higher-order sensory substitution, the system conveys navigational information through adaptive vibrotactile feedback.

• Modular vibration units operate independently for spatial feedback
• Open path mode guides users toward the most traversable direction
• Depth mode maps proximity to vibration intensity
• User testing with participants with blindness or low vision

UNAV is a vision-based localization and navigation pipeline designed for end users with blindness and low vision. Given a query image captured in a mobile application, the system performs visual place recognition to retrieve similar reference images and estimates the user's location via a weighted-average approach. It also estimates user heading using a perspective-n-point method based on 2D–3D correspondences, then computes a shortest path on a navigable map to support end-to-end wayfinding.

• Visual place recognition retrieves similar images from a reference database
• Location estimated using a weighted-average method on matched image geolocations
• Heading estimated via PnP using 2D–3D point correspondences
• Topometric map built by projecting a 3D sparse map onto a 2D floor plan
• Shortest path computed with Dijkstra's algorithm on a navigable map

Curb Detector is a vision-based wearable assistive system designed to help people who are blind or have low vision safely detect and orient to curbs in urban environments. Using an RGB camera and an embedded processing platform, the system segments curbs in real time and provides early warning and orientation information before the user reaches an elevation change.

• Real-time curb segmentation using a YOLOv8 model trained on a custom dataset
• Adaptive auditory feedback including beeps, abstract sonification, and speech
• Advanced warning with a larger safety window than a white cane
• Provides curb distance and orientation information

Construction Site Detection is a real-time, computer-vision–based assistive system designed to help people who are blind or have low vision identify temporary construction hazards in urban environments. The system provides advance warning of construction zones that disrupt familiar routes and introduce safety risks such as uneven surfaces, barriers, and altered walkways.

• Open-vocabulary detection of diverse construction elements
• YOLO-based model specialized for scaffolding and poles
• OCR interprets construction signage
• Validated in static sites and dynamic walking routes

KineReach is a high-precision, time-synchronized measurement and analysis pipeline for quantifying visuomotor coordination during goal-directed behavior. Using controlled paradigms that dissociate gaze from reach planning, the platform models eye–hand timing, variability, and online correction to reveal mechanistic signatures of sensorimotor function, impairment, and recovery.

• High-precision synchronization of gaze, limb kinematics, and task events
• Paradigms separating planning from execution
• Trial-level eye–hand latency and coordination variability
• Mechanistic biomarkers responsive to rehabilitation

This project investigates neural-muscular coupling as a biomarker for motor recovery following neurological events. By fusing brain and muscle activity signals, we construct connectivity networks and extract graph-theoretic features to capture recovery trajectories beyond conventional clinical scales.

• Fused brain-muscle connectivity network construction
• Graph-theoretic biomarkers for recovery profiling
• Correlation with functional and kinematic outcomes
• Cross-sectional and longitudinal study design

This project enhances standardized clinical hand-function assessments by integrating multimodal sensing pipelines. We derive objective biomarkers related to movement strategy, quality, and error patterns—going well beyond traditional completion-time measures to support sensitive phenotyping and rehabilitation-focused evaluation.

• Synchronized gaze and motion capture during functional tasks
• Biomarkers beyond completion time
• Subtask-level event timing and kinematics
• Quantitative outcomes for longitudinal tracking

This project augments standard clinical dexterity assessments with advanced computer vision pipelines to objectively quantify fine motor performance in individuals with neurological conditions. By fusing object detection with hand tracking, the system produces granular performance metrics beyond simple completion time.

• Augments standard assessments using vision-based pipelines
• Fusion of object detection and eye-hand tracking
• Quantifiable, repeatable datasets
• Robust detection and error handling

This project develops a markerless, multi-camera pipeline for reconstructing 3D hand kinematics during dexterity tasks in children. Using neural network-based pose estimation and multi-view reconstruction, it captures fine finger movements without wearable sensors, overcoming key limitations of traditional motion capture. Paired with modern eye-tracking, this framework supports more precise diagnostics and targeted rehabilitation for children with neurologic injury.

• Markerless multi-camera 3D hand reconstruction
• Task-specific neural network training for pediatric hands
• Millimeter-scale spatial reconstruction without sensors
• Compatibility with integrated eye-tracking systems