Joint 5D Pen Input for Light Field Displays

James Tompkin, Samuel Muff, James McCann, Hanspeter Pfister,
Jan Kautz, Marc Alexa, Wojciech Matusik
ACM User Interface Software and Technology (UIST 2015)

Abstract

Light field displays allow viewers to see view-dependent 3D content as if looking through a window; however, existing work on light field display interaction is limited. Yet, they have the potential to parallel 2D pen and touch screen systems which present a joint input and display surface for natural interaction. We propose a 4D display and interaction space using a dual-purpose lenslet array, which combines light field display and light field pen sensing, and allows us to estimate the 3D position and 2D orientation of the pen. This method is simple and fast (150 Hz), with position accuracy of 2–3 mm and precision of 0.2–0.6 mm from 0–350 mm away from the lenslet array, and orientation accuracy of 2° and precision of 0.2–0.3° within 50°. Further, we 3D print the lenslet array with embedded baffles to reduce out-of-bounds cross-talk, and use an optical relay to allow interaction behind the focal plane. We demonstrate our joint display/sensing system with interactive light field painting.

Video

Video (MP4 82MB)

Files

Paper
(PDF 13MB) 		Presentation
(PPTX 83MB) 		Lenslet Sheet
(STL 70MB) 		Lenslet Sheet Sample
(STL 1MB)

Bibtex

@inproceedings{Tompkin:2015:UIST,
    author = {James Tompkin and Samuel Muff and James McCann
              and Hanspeter Pfister and Jan Kautz
              and Marc Alexa and Wojciech Matusik},
    title = {Joint 5D Pen Input for Light Field Displays},
    booktitle = {The 28th Annual ACM Symposium on User Interface
                 Software and Technology, UIST'15},
    month = nov,
    year = {2015},
}

Acknowledgements

Thank you to Olivier Bau, Daniel Haehn, Xavier Snelgrove, and Stanislav Jakuscievski for their hard work and support; Maryam Pashkam and Ken Nakayama for the Polhemus Fastrak; and Lara Booth for her artistry. We thank the (New) Stanford Light Field Archive for the photographic light fields, the Stanford 3D Scanning Repository for the bunny model, and I-R Entertainment Ltd., Morgan McGuire, and Guedis Cardenas for the human head model. Marc Alexa thanks support from grant ERC-2010-StG 259550 (“XSHAPE”), and James Tompkin and Hanspeter Pfister thank NSF CGV-1110955.