A Lightweight and Efficient System for Tracking Handheld Objects in Virtual Reality

Ya-Kuei Chang
National Tsing Hua University
Jui-Wei Huang
National Tsing Hua University
Chien-Hua Chen
National Taiwan University of Science and Technology
Chien-Wen Chen
National Cheng Kung University and Academia Sinica
Jian-Wei Peng
National Cheng Kung University
Min-Chun Hu
National Cheng Kung University
Chih-Yuan Yao
National Taiwan University of Science and Technology
Hung-Kuo Chu
National Tsing Hua University


While the content of virtual reality (VR) has grown explosively in recent years, the advance of designing user-friendly control interfaces in VR still remains a slow pace. The most commonly used device, such as gamepad or controller, has fixed shape and weight, and thus can not provide realistic haptic feedback when interacting with virtual objects in VR. In this work, we present a novel and lightweight tracking system in the context of manipulating handheld objects in VR. Specifically, our system can effortlessly synchronize the 3D pose of arbitrary handheld objects between the real world and VR in realtime performance. The tracking algorithm is simple, which delicately leverages the power of Leap Motion and IMU sensor to respectively track object’s location and orientation. We demonstrate the effectiveness of our system with three VR applications use pencil, ping-pong paddle, and smartphone as control interfaces to provide users more immersive VR experience.


Hardware components:

        (a) A Leap Motion sensor.

        (b) An IMU sensor.

        (c) An inductive tray which is composed of a Vive         tracker and a pair of RFID tag/sensor.

        (d) An optionaltouch sensor attached to the hand-held         object for offering extra inputs.