KrishnaVision: A Multimodal Virtual Interface  Combining MediaPipe-Hands Optimization and Gemini  AI for Context- Aware HCI

Q: How many days will it take for my paper to be published?

The review time for papers is not fixed. However, if the paper is accepted and the author completes the processing charges formalities, the paper will be published within a few working days.

Q: I would like to receive a hard copy of the journal materials. Are there any additional charges?

You can log in to the author portal and pay 500 INR to receive the hard copy materials.

Rupkatha De; Aritro Saha

KrishnaVision: A Multimodal Virtual Interface Combining MediaPipe-Hands Optimization and Gemini AI for Context- Aware HCI

Authors: Rupkatha De, Aritro Saha

Unique Paper ID: 180711
Volume: 12
Issue: 1
PageNo: 2909-2917

Keywords: Adaptive HCI MediaPipe Optimization Multimodal Fusion Gemini AI Gesture-Voice Integration

Abstract:
This work introduces Krishna Vision, an innovative virtual mouse system that is synergistically bringing together MediaPipe's hand tracking and Gemini's multimodal AI to develop a human-computer interface that can adapt. Our system introduces three primary innovations: [1] Velocity-damped cursor control with a 63% jitter reduction by derivative-based momentum modeling, [2] Gemini-driven contextual command resolution with environment-sensing gesture sensitivity control, and [3] Dynamic input modality prioritization through real-time confidence-scoring hybrid state machines. Results from benchmarks achieve 97.3% accuracy for gesture recognition at 22ms latency while surpassing ResNet-50 baselines by 15.2% with 41% reduced power usage. Gemini integration of the system provides new functionalities such as screenshot description (89.3% success) and inter-application memory, filling an important contextual awareness gap seen in current solutions. Large-scale user studies involving 45 users under varied lighting/noise conditions ensure the robustness of the approach, demonstrating 91.5% success on complicated hybrid commands.

Download article

email to a friend

Cite This Article

ISSN: 2349-6002
Volume: 12
Issue: 1
PageNo: 2909-2917

KrishnaVision: A Multimodal Virtual Interface Combining MediaPipe-Hands Optimization and Gemini AI for Context- Aware HCI

Available:https://ijirt.org/article?manuscript=180711

Impact Factor
8.01 (Year 2024)

UGC Approved
Journal no 47859

Join Our IPN

IJIRT Partner Network

Submit your research paper and those of your network (friends, colleagues, or peers) through your IPN account, and receive 800 INR for each paper that gets published.

Join Now