Yao Lu and Walterio Mayol-Cuevas
ISMAR 2019 Adjunct Proceedings, ISMAR, Beijing, 2019.
In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.
