Paper Reading #3: HoloDesk: Direct 3D Interactions with a Situated SeeThrough Display

HoloDesk: Direct 3D Interactions with a Situated SeeThrough Display

Otmar Hilliges (1), David Kim (1,2), Shahram Izadi (1), MalteWeiss (1,3), Andrew D.Wilson (4)

1 Microsoft Research   2 Culture Lab               3 RWTH Aachen University   4 Microsoft Research

 7 JJ Thomson Ave         Newcastle University,   52056 Aachen,                       One Microsoft Way

 Cambridge, UK             Newcastle, UK             Germany                                Redmond, WA

{otmarh,b-davidk,shahrami,awilson}@microsoft.com, weiss@cs.rwth-aachen.de

Author Bios:

Otmar Hilliges

• Part of the Computer Mediated Living Group at Microsoft Research in Cambridge, UK.

• PhD in Compter Science from Ludwig-Maximilians Universität München / LMU Munich

David Kim

• Part of Microsfot Research in Cambridge, UK

Shahram Izadi

• Research scientist at Microsoft Research Cambridge. 

• Xerox PARC before that.

• PhD with Tom Rodden and Yvonne Rogers working on the EQUATOR project.

Malte Weiss

• PhD student at the Media Computing Group of RWTH Aachen University.

Andrew D.Wilson

• Principal researcher at Microsoft Research

• MS and PhD at the MIT Media Laboratory.

Summary:

A user can interact with 3D virtual objects that act like physical objects through pushing, scooping up and even grasping the virtual objects within the scene of the HoloDesk. The creators made manipulating the virtual objects as intuitive as possible by being able to grasp objects and turning and rotating them. As shown in the pictures below, physical objects can also be used to interact with the virtual objects.  Of course it is limited by the fact that the virtual objects cannot interact back with the physical objects, they can only react as a physical object would. Physical objects even cast virtual shadows on virtual objects that are bellow them.

It consists of an interaction volume where the virtual objects appear to be when looking through the glass transparent glass beamsplitter right above the volume.  A RGB Webcam measures the orientation and position of the user's head so that the projected 3D volume from the LCD is displayed so that when the user looks through the beamsplitter it is a if the projection is under it. A Kinect is used to track the user's hands and other physical objects that may interact with the virtual objects in the volume. The RGB Webcam is continuously updating where the user's head is and refreshing the projection to keep up the illusion of the virtual 3D volume.

Related work not referenced in the paper:

• Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System
• Talks about the mathematics needed in order to successfully display an augmented reality such as the GPS coordinates and orientations of users and virtual objects. It does not use a webcam to judge orientation or a beamsplitter to give the illusion of a 3D space like the HoloDesk does.
  
•  Face to Face Collaborative AR on Mobile Phones
• From the massive amount of augmented reality already being used in phones, they went a step further and added a face to face aspect. When connected to another person's phone they can see each other as if the other player was on the other side of a table tennis game. This was limited to a 2D representation since it was only using a camera phone where as the HoloDesk is fully a 3D experience.
• Table Top Augmented Reality System for Conceptual Design and Prototyping
• This is very similar to the concept of the HoloDesk, but was implemented in a very different way. A large LCD screen is lifted a couple inches off a table and a person reaches under it where a camera displays your hands on the LCD screen along with the 3D environment to manipulate. you cannot directly see your hands and so is not as natural looking as the HoloDesk is. The virtual objects also do not act as they would if they were real as they attempt to do in HoloDesk.
• The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays
• The office of the future uses magnetism to track a users head orientation to project a seemingly 3D image on any surface instead of a camera and does not allow direct manipulation of virtual objects like HoloDesk does.
• Combining Multiple Depth Cameras and Projectors for Interactions On, Above, and Between Surfaces
• LightSpace is a small room with projectors and a depth camera suspended on the ceiling. The depth camera follows a person's movements and registers gestures to interact with the walls and table. It does not display an interactive 3D environment, but it does project 2D images into the user's hand or the floor.
• MirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop 
• The MirageTable is very similar to the HoloDesk in that a camera follows the user's head and virtual objects can be manipulated with just your hands. However, objects cannot be grasped and there is a projector instead of an LCD screen and beamsplitter.
• Vision-based 3D Finger Interactions for Mixed Reality Games with Physics Simulation
• Uses a laptop and 3D camera to interact with virtual objects that are displayed on the laptop's screen. The only way to interact with the objects is by selecting them with your fingertip at which point a virtual tether links the object to your finger. There is no 3D volume and the interactions are not as intuitive as the HoloDesk.
• Interactions in the Air: Adding Further Depth to Interactive Tabletops
• Uses a projector under the table to display virtual objects that can be grabbed by pinching your fingers above the table.  Virtual objects cannot be picked up and the display is 2D so it does not require head tracking.
• Multimodal Interaction in an Augmented Reality Scenario
• A headset and glasses creates an augmented reality where a depth camera identifies objects and can follow the user's fingers in order to select menu items and physical objects by projecting a line along a pointing finger. There are no virtual objects to manipulate and is a mobile device unlike the HoloDesk.
• Simulating Educational Physical Experiments in Augmented Reality
• A head mounted unit creates an augmented reality through glasses where a pen with white balls attached for a depth camera to follow can create virtual objects and give them properties by drawing on a pad that is also being tracked with white balls attached to it. All virtual interaction is from the pen and pad where as the HoloDesk does not use any tools for interaction.

Evaluation:

There was an informal and a formal evaluation done. The informal evaluation simply had hundreds of users play with it without any prior instruction or objective to do.  All observations were qualitative and subjective based on the observer. The formal evaluation used a simple task to measure accuracy of where the virtual objects appear to be with three display types. standard setup(DHD), standard setup with stereo output(SHD) and Nvidia 3D Vision LCD shutter glasses(IHD).  The task started with a red cube that when touched would randomly spawn another cube and time how long it took the user to touch the new cube. when the second cube was touched, another appeared waiting to be touched. The results are displayed below.

Discussion:
I found the idea of the HoloTable to not be very novel, but how they went about creating it absolutely was.  There were many other augmented reality tables, but none had such intuitive interaction with the virtual objects as the HoloDesk.  I do not think the evaluation fully tested every aspect of the device, but they got very good reviews from the people how participated in the informal evaluation.  I think that it is a step up from all the other augmented reality tables out there.

Paper Reading #2: Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects

Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects

Munehiko Sato (1,2), Ivan Poupyrev (1), Chris Harrison (1,3)

1 Disney Research Pittsburgh,     2  Graduate School of Engineering,  3 HCI Institute,

   4720 Forbes Avenue,                   The University of Tokyo,                Carnegie Mellon University,

   Pittsburgh, PA 15213 USA           Hongo 7-3-1, Tokyo                     5000 Forbes Avenue,

   {munehiko.sato, ivan.poupyrev}    113-8656 Japan                            Pittsburgh, PA 15213 USA

   @disneyresearch.com                                                                  chris.harrison@cs.cmu.edu

Author Bios:

Munehiko Sato

• PhD candidate at Department of Advanced Interdisciplinary Studies, Graduate School of Engineering at the University of Tokyo.

• working toward a PhD degree in the Cyber Interface Lab (Hirose Tanikawa Lab) under the supervision of Prof. Michitaka Hirose.

Ivan Poupyrev

• Senior research scientist at Walt Disney Research.

• Worked as a researcher at Sony Computer Science Laboratories and the Advanced Telecommunication Research Institute International.

• Ph.D. dissertation in Hiroshima University, Japan.

Chris Harrison

• Ph.D. candidate in the Human-Computer Interaction Institute at Carnegie Mellon University advised by Scott Hudson.

Summary:

Their Touché circuit device uses their new found Swept Frequency Capacitive Sensing technique to register multiple types of touches from the human body.  The device can be attached into any conductive object or material. Conductive diodes can be retrofitted to any non-conductive object touch sensitivity would be found useful to have. The human body can manipulate small electrical signals and Touché can monitor these changes.  The main novel idea is to measure the electrical alterations from human contact at different frequencies in a "sweeping" fashion.  This is a new concept because it was not feasible without the small and powerful processors that are now available. By taking data points at different frequencies, the sensitivity of the touch sensor is greatly improved.  In the figure to the right, picture a displays wrist bands that sense hand gestures in order to control a phone. Picture b shows a door knob sensitive to different types of touches. Picture c shows water that can sense how much of a person's body is in contact with he water. Picture d has a smart phone with a casing that is capable of registering exactly how a person is holding the phone and can respond accordingly.  These are only a few of the possible uses for Touché.  It is a very simple to install into a device. Only a single wire is needed to be in contact with the conductive surface for it to work properly and it is relatively cheap to create.

Related work not referenced in the paper:

• ReachMedia: On-the-move interaction with everyday objects 
• This is a wrist only device that uses radio waves to detect objects the person is holding. The object requires a RFID tag for the wrist device to detect it and give the user helpful information about it as they pick it up.  Touché is not used for this purpose and uses electrical human touch sensors not radio waves.
•  Enabling mobile micro-interactions with physiological computing
• Has an arm band that registers muscle movements in the hand to control a computer or other device. Also uses "Skinput" which is a touch UI from a projection onto the skin. No electrical signals are sensed like with the Touché.
• Multimodal Human Computer Interaction: A Survey
• States possible touch sensitive or gesture sensing ways of collecting commands by objects like computers. Does not say much about how to actually collect the data like Touché.
• Evaluating Capacitive Touch Input on Clothes
• Uses micro buttons sewn into fabrics in a visually appealing way with wires that can be stretched as to keep the fabric flexible.  The creators of Touché want to get away from actual buttons and create objects in themselves responsive.
• gRmobile: A Framework for Touch and Accelerometer Gesture Recognition for Mobile Games
• Uses an accelerometer and camera to visually register gestures and orientation. This is a completely different way of collecting gestures and their way is not very novel.
• Your Noise is My Command: Sensing Gestures Using the Body as an Antenna
• I found this one very interesting because it uses the EM fields already generated by wires in a home to be able to map where and what a person is touching such as a wall. This is similar but no the same as Touché since it uses EM instead of electrical inductance in the human body.
•  Sensing Foot Gestures from the Pocket 
• Attempts to provide silent commands to a phone in your pocket by sensing the orientation of the phone and flexing of the toes.  It uses buttons inside of a shoe to double tap or scrolling motion. I see many problems with this design. If you are running there is no telling what the sensors are going to accidentally register.
• PACER: Fine-grained Interactive Paper via Camera-touch Hybrid Gestures on a Cell Phone
• Uses a camera phone to visually map a physical piece of paper and a virtual one on the phone. You can then do things like search a term simply by selecting it on the virtual picture in the phone. This a completely different way of interacting with objects from Touché.
•  On Body Capacitive Sensing for a Simple Touchless User Interface
• Treats the human body one plate of a capacitor and senses guestures without actually touching anything. The motivation came from doctors not wanting to touch anything in fear of contamination. Where this uses capacitance Touché uses inductance of the entire body.
• PocketTouch: Through-Fabric Capacitive Touch Input
• Uses capacitance just like a touch screen on a phone, but it is capable of being put into many different types of fabric of eyes free use of your phone while it is in your pocket. This uses capacitance with the human body instead of inductance of an electrical signal like Touché.

Evaluation:

They evaluated the accuracy of Touché to correctly register different types of gestures with five different objects and materials: a doorknob, a table, a phone case, on-body sensing from wrist bands, and water.  They used two groups of 12 participants each. The groups were shown pictures of the possible gestures they could make with each object. They then were told to make these gestures one at a time. What Touché actually registered each of their attempted gestures as was kept from the participant and experimenters until the end. In the first group, experimenters used data gathered from each participant at the beginning of their testing in order to fine tune the Touché devices to the specific touch of each participants. The second gorup was "walk-up" where there was no prior data collection to their testing. The graphs below show their findings. all of their data collection was quantitative and objective.  More data was collected for the "walk-up" group because it was done on a later date.

Discussion:

I found this paper to be very interesting and can I can easily see this technology in our immediate future. the Touché device can potentially be attached to any object. It is so small and simple in its design that it can be added to everyday items cheaply without anyone even being aware it is there. Know one will notice a normal looking, key less door that will only open when you use the correct combination of hand gestures.

Paper Reading #1: Portico: Tangible Interaction on and around a Tablet

Portico: Tangible Interaction on and around a Tablet


Daniel Avrahami (1,2),  Jacob O. Wobbrock (2) and  Shahram Izadi (3)

1 Intel                                         2 The Information School | DUB Group   3 Microsoft Research
   2200 Mission College Blvd.        University of Washington                         7 J J Thomson Avenue
   Santa Clara, CA 95054-1549      Seattle, WA 98195-2840                         Cambridge, UK

Author Bios:

Daniel Avrahami

• Senior researcher at Intel.
• Ph.D. and M.Sc. in Human-Computer Interaction from the HCI Institute (HCII) at the School of Computer Science (SCS) at Carnegie Mellon University (CMU).

Jacob O. Wobbrock

• Associate Professor in the Information School and an Adjunct Associate Professor in the Department of Computer Science & Engineering at the University of Washington.
• Ph.D. from the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University.

Shahram Izadi

• Research scientist at Microsoft Research Cambridge.
• Xerox PARC before that.
• PhD with Tom Rodden and Yvonne Rogers working on the EQUATOR project.

Summary: 

They improved a tablet computer's tangible user interface (TUI)  by attaching two cameras and software that can identify and track specific objects.  They call it Portico because it extends the tablet's interactive possibilities without compromising the portability. In the picture, an application called Tabletgotchi is tracking a stuffed animal and can tell that it is "drinking" from the water displayed on the tablet's screen by registering its location and orientation from the raised cameras. The tablet can respond in a seemingly natural way to how the user moves the zebra. 

Related work not referenced in the paper:

• Tangible Chess for the Compaq Tablet PC
• uses a tablet as a chess board. Each chess piece is tracked by contact with the touch screen where Portico  uses cameras to track objects instead.
• Potentials of the Tangible User Interface (TUI)  in Enhancing Inclusion of People with Special Needs in the ICT-Assisted Learning and e-Accessibility
• Explains how tangible user interfaces, such as Portico can be used to teach easily because of its natural way of interacting.
• TouchPlanVS Lite 
• A Tablet-based Tangible Multitouch Planning System for Virtual TV Studio Productions - Uses a tablet to control an array of televisions and does not use cameras to track objects.
• The Tangible Pathfinder Design of a Wayfinding Trainer for the Visually Impaired 
• It is theoretical in explaining how a virtual reality that uses tangible surfaces to help a visually impaired patient find their way around.
• Kolab: Appropriation & Improvisation in Mobile Tangible Collaborative Interaction 
• A Microsoft Kinect camera is used to track objects on a flat surface. It differs from the Portico because it is completely reliant on the camera for input instead of using a tablet in conjunction with the camera.
• TUIC: Enabling Tangible Interaction on Capacitive Multi-touch Display 
• An spatial tags are used on an iPad to register a location's position on the screen. Unlike Portico, it is restricted to the iPad's screen for interaction.
• Tangible Interaction in Mixed Reality Application
• Explains the importance TUI devices will have in the future and describes many examples, but none that I quickly read were exactly like Portico.
• Tangible Remote Controllers for Wall-Size Displays
• Uses a tablet to help users explore displays that are too large to see all at once such as a wall of screens. This is not the same idea as Portico. 
• Tangible User Interface for Chemistry Education: Comparative Evaluation and Re-Design 
• The authors of this paper hope that a TUI device could replace the ball and stick type models of chemicals to better teach students while keeping as much of the physical intuitiveness of the old way of teaching as possible. 
• Tangible Bits: Beyond Pixels
• Explains contributions by the Tangible Media Group and others which do not appear to be similar to Portico.

All of these related work either show their idea for a TUI type design or talk about many ways that it can be used. They all have a unique idea about the subject and many of them even reference each other. All of the papers see TUI devices as a potentially huge teaching tool or a more natural way of design. Portico is a unique idea as I have not found anything similar in the papers above.

Evaluation:

They evaluated their work by creating a few applications that each use a different aspect of Portico for users to test. The applications were simple games that were tested informally by three children. A four year old and two three year olds. They used the qualitatively and subjectively observed how the children played with Portico to evaluate it. They took note of what the children had trouble with, what was not as intuitive as they had thought and their behavior as they played. The applications evaluated how Portico can track objects and then act accordingly to make the transition from the physical world and the virtual world as seamless as possible. They also subjectively observed that the range of the cameras could be increased to further its interactive surface. The only quantitative and objective evaluation taken was the frame rate of the cameras. They found that the cameras ran from 18 fps to 30 fps depending on the amount of objects in view of the cameras.

Discussion:

I believe that Portico is a step forward in TUI devices. There were no other papers I could find that went about using cameras and a tablet in a similar way. Many used tablets as a way to manipulate a larger display such as a TV instead of adding onto the tablet itself. Portico is absolutely a novel idea. The evaluation covered exactly what Portico was meant for and exposed some faults that they then explained solutions to.

A little about me

• nrubenstein24@gmail.com
• 4th year Senior
• I am taking this class because I believe for software to be great and last a long time it must be user friendly.
• I bring programming experience with multiple languages and some knowledge of psychology to the class.
• My professional life goals are to program in a welcoming environment where I can continue to learn and become a better programmer throughout my life and to always have a new challenge to keep me sharp.
• My personal life goals are to never become tired of programming and learn as much as I can.
• I have a job offer that I have accepted beginning after I graduate. I would like to travel Europe before I start working and it is too late.
• I expect to be working as a developer in the oil industry in ten years.
• I think the next biggest technological advancement in computer science will be the synchronization and integration of all technology to further increase the efficiency of our society. 
• If I could travel back in time, I would like to meet Albert Einstein as a child so I could see what sort of kid turned into one of the greatest minds in history.
• My favorite shoes are Pumas because I have very thin and long feet and they are one of the only shoes that fit me well.
• I would learn Mandarin Chinese  because I find Chinese culture incredibly interesting and want to travel there sometime.
• I am left handed and can juggle.