Paper Reading #3: HoloDesk: Direct 3D Interactions with a Situated SeeThrough Display

HoloDesk: Direct 3D Interactions with a Situated SeeThrough Display

Otmar Hilliges (1), David Kim (1,2), Shahram Izadi (1), MalteWeiss (1,3), Andrew D.Wilson (4)

1 Microsoft Research   2 Culture Lab               3 RWTH Aachen University   4 Microsoft Research

 7 JJ Thomson Ave         Newcastle University,   52056 Aachen,                       One Microsoft Way

 Cambridge, UK             Newcastle, UK             Germany                                Redmond, WA

{otmarh,b-davidk,shahrami,awilson}@microsoft.com, weiss@cs.rwth-aachen.de

Author Bios:

Otmar Hilliges

• Part of the Computer Mediated Living Group at Microsoft Research in Cambridge, UK.

• PhD in Compter Science from Ludwig-Maximilians Universität München / LMU Munich

David Kim

• Part of Microsfot Research in Cambridge, UK

Shahram Izadi

• Research scientist at Microsoft Research Cambridge. 

• Xerox PARC before that.

• PhD with Tom Rodden and Yvonne Rogers working on the EQUATOR project.

Malte Weiss

• PhD student at the Media Computing Group of RWTH Aachen University.

Andrew D.Wilson

• Principal researcher at Microsoft Research

• MS and PhD at the MIT Media Laboratory.

Summary:

A user can interact with 3D virtual objects that act like physical objects through pushing, scooping up and even grasping the virtual objects within the scene of the HoloDesk. The creators made manipulating the virtual objects as intuitive as possible by being able to grasp objects and turning and rotating them. As shown in the pictures below, physical objects can also be used to interact with the virtual objects.  Of course it is limited by the fact that the virtual objects cannot interact back with the physical objects, they can only react as a physical object would. Physical objects even cast virtual shadows on virtual objects that are bellow them.

It consists of an interaction volume where the virtual objects appear to be when looking through the glass transparent glass beamsplitter right above the volume.  A RGB Webcam measures the orientation and position of the user's head so that the projected 3D volume from the LCD is displayed so that when the user looks through the beamsplitter it is a if the projection is under it. A Kinect is used to track the user's hands and other physical objects that may interact with the virtual objects in the volume. The RGB Webcam is continuously updating where the user's head is and refreshing the projection to keep up the illusion of the virtual 3D volume.

Related work not referenced in the paper:

• Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System
• Talks about the mathematics needed in order to successfully display an augmented reality such as the GPS coordinates and orientations of users and virtual objects. It does not use a webcam to judge orientation or a beamsplitter to give the illusion of a 3D space like the HoloDesk does.
  
•  Face to Face Collaborative AR on Mobile Phones
• From the massive amount of augmented reality already being used in phones, they went a step further and added a face to face aspect. When connected to another person's phone they can see each other as if the other player was on the other side of a table tennis game. This was limited to a 2D representation since it was only using a camera phone where as the HoloDesk is fully a 3D experience.
• Table Top Augmented Reality System for Conceptual Design and Prototyping
• This is very similar to the concept of the HoloDesk, but was implemented in a very different way. A large LCD screen is lifted a couple inches off a table and a person reaches under it where a camera displays your hands on the LCD screen along with the 3D environment to manipulate. you cannot directly see your hands and so is not as natural looking as the HoloDesk is. The virtual objects also do not act as they would if they were real as they attempt to do in HoloDesk.
• The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays
• The office of the future uses magnetism to track a users head orientation to project a seemingly 3D image on any surface instead of a camera and does not allow direct manipulation of virtual objects like HoloDesk does.
• Combining Multiple Depth Cameras and Projectors for Interactions On, Above, and Between Surfaces
• LightSpace is a small room with projectors and a depth camera suspended on the ceiling. The depth camera follows a person's movements and registers gestures to interact with the walls and table. It does not display an interactive 3D environment, but it does project 2D images into the user's hand or the floor.
• MirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop 
• The MirageTable is very similar to the HoloDesk in that a camera follows the user's head and virtual objects can be manipulated with just your hands. However, objects cannot be grasped and there is a projector instead of an LCD screen and beamsplitter.
• Vision-based 3D Finger Interactions for Mixed Reality Games with Physics Simulation
• Uses a laptop and 3D camera to interact with virtual objects that are displayed on the laptop's screen. The only way to interact with the objects is by selecting them with your fingertip at which point a virtual tether links the object to your finger. There is no 3D volume and the interactions are not as intuitive as the HoloDesk.
• Interactions in the Air: Adding Further Depth to Interactive Tabletops
• Uses a projector under the table to display virtual objects that can be grabbed by pinching your fingers above the table.  Virtual objects cannot be picked up and the display is 2D so it does not require head tracking.
• Multimodal Interaction in an Augmented Reality Scenario
• A headset and glasses creates an augmented reality where a depth camera identifies objects and can follow the user's fingers in order to select menu items and physical objects by projecting a line along a pointing finger. There are no virtual objects to manipulate and is a mobile device unlike the HoloDesk.
• Simulating Educational Physical Experiments in Augmented Reality
• A head mounted unit creates an augmented reality through glasses where a pen with white balls attached for a depth camera to follow can create virtual objects and give them properties by drawing on a pad that is also being tracked with white balls attached to it. All virtual interaction is from the pen and pad where as the HoloDesk does not use any tools for interaction.

Evaluation:

There was an informal and a formal evaluation done. The informal evaluation simply had hundreds of users play with it without any prior instruction or objective to do.  All observations were qualitative and subjective based on the observer. The formal evaluation used a simple task to measure accuracy of where the virtual objects appear to be with three display types. standard setup(DHD), standard setup with stereo output(SHD) and Nvidia 3D Vision LCD shutter glasses(IHD).  The task started with a red cube that when touched would randomly spawn another cube and time how long it took the user to touch the new cube. when the second cube was touched, another appeared waiting to be touched. The results are displayed below.

Discussion:
I found the idea of the HoloTable to not be very novel, but how they went about creating it absolutely was.  There were many other augmented reality tables, but none had such intuitive interaction with the virtual objects as the HoloDesk.  I do not think the evaluation fully tested every aspect of the device, but they got very good reviews from the people how participated in the informal evaluation.  I think that it is a step up from all the other augmented reality tables out there.