Phonetroller: Visual Representations of Fingers for Precise Touch Input when using a Phone in VR
Fabrice Matulic, Aditya Ganeshan, Hiroshi Fujiwara, Daniel Vogel
CHI'21: ACM CHI Conference on Human Factors in Computing Systems
Session:
AR and VR - B
AR and VR - C
Abstract
Smartphone touch screens are potentially attractive for interaction in virtual reality (VR). However, the user cannot see the phone or their hands in a fully immersive VR setting, impeding their ability for precise touch input. We propose mounting a mirror above the phone screen such that the front-facing camera captures the thumbs on or near the screen. This enables the creation of semi-transparent overlays of thumb shadows and inference of fingertip hover points with deep learning, which help the user aim for targets on the phone. A study compares the effect of visual feedback on touch precision in a controlled task and qualitatively evaluates three example applications demonstrating the potential of the technique. The results show that the enabled style of feedback is effective for thumb-size targets, and that the VR experience can be enriched by using smartphones as VR controllers supporting precise touch input.
DOI:: [ Ссылка ]
WEB:: [ Ссылка ]
Video Previews of CHI 2021
