Immersive Future Media Technologies: From 3D Video to Sensory Experience

--tutorial to be held at ACM Multimedia 2010, Oct 25th, Morning, Florence, Italy

--download as PDF

For registration details, please consult ACM Multimedia 2010 Web site!

Instructors: Karsten Müller (Fraunhofer/Heinrich-Hertz-Institut, Berlin) and Christian Timmerer (Klagenfurt University, Austria)

Abstract: The past decade has witnessed a significant increase in the research efforts around the Quality of Experience (QoE) which is generally referred to as a human-centric paradigm for the Quality of a Service (QoS) as perceived by the (end) user. As it puts the end user in the center stage, it may have various dimensions and one dimension recently gained momentum is 3D video. Another dimension aims at going beyond 3D and promises advanced user experience through sensory effects, both introduced briefly in the following.

3D Video: Stereo and Multi-View Video Technology
3D related media technologies have recently developed from pure research-oriented work towards applications and products. 3D content is now being produced on a wider scale and first 3D applications have been standardized, such as multi-view video coding for 3D Blu Ray disks. This part of the tutorial starts with an overview on 3D in the form of stereo video based systems, which are currently being commercialized. Here, stereo formats and associated coding are introduced. This technology is used for 3D cinema applications and mobile 3DTV environments. For the latter, user requirements and profiling will be introduced as a form to assess user quality of experience. For 3D home entertainment, glasses-free multi-view displays are required, as more than one user will watch 3D content. For such displays, the current stereo solutions need to be extended. Therefore, new activities in 3D video are introduced. These 3D solutions will develop a generic 3D video format with color and supplementary geometry data, e.g. depth maps, and associated coding and rendering technology for any multi-view display, independent of the number of views. As such technology is also developed in international consortia, the most prominent, like the 3D@HOME consortium, the EU 3D, Immersive, Interactive Media Cluster and the 3D video activities in ISO-MPEG are introduced.

Advanced User Experience through Sensory Effects
This part of the tutorial addresses a novel approach for increasing the user experience – beyond 3D – through sensory effects. The motivation behind this work is that the consumption of multimedia assets may stimulate also other senses than vision or audition, e.g., olfaction, mechanoreception, equilibrioception, or thermoception that shall lead to an enhanced, unique user experience. This could be achieved by annotating the media resources with metadata (currently defined by ISO/MPEG as part of the MPEG-V standard) providing so-called sensory effects that steer appropriate devices capable of rendering these effects (e.g., fans, vibration chairs, ambient lights, perfumer, water sprayers, fog machines, etc.). In particular, we will review the concepts and details of the forthcoming MPEG-V standard and present our prototype architecture for the generation, transport, decoding and use of sensory effects. Furthermore, we will present details and results of a series of formal subjective quality assessments which confirm that the concept of sensory effects is a vital tool for enhancing the user experience.

Course Outline
Introduction and Overview (~10min.)
3D Video: Stereo and Multi-View Video Technology (~90min)

• User Requirements for 3D video technologies
• Stereo Video solutions for 3D cinema and mobile applications
• User experience and profiling for mobile 3DTV
• Format description and coding for stereo and multi-view video technology
• Depth-enhanced 3D video solutions for 3D home entertainment

Advanced User Experience through Sensory Effects (~90min)

• MPEG-V: context and objectives including an overview of all parts.
• In depth review of Part 3 of MPEG-V entitled Sensory Information.
• A test-bed for the quality of multimedia experience evaluation of sensory effects and demonstration.
• How to improve the Quality of Experience through sensory effects? Results from first subjective experiments.
• Conclusions and future work.

Intended Audience
Intermediate, specifically, graduate students and researchers interested in 3D video and sensory experiences.

Biography of Presenters

Karsten Müller received the Dipl.-Ing. and Dr.-Ing. degree from the Technical University of Berlin, Germany, in 1997 and 2006 respectively. In 1993 he spent one year studying Electronics and Communication Engineering at Napier University of Edinburgh/Scotland, including a half-year working period at Integrated Communication Systems Inc. in Westwick near Cambridge/England. In this working period he developed software for voice mail systems and statistical analysis of caller data. In 1996 he joined the Heinrich-Hertz-Institute (HHI) Berlin, Image Processing Department, were he is a project coordinator for European projects in the field of 3D media technology. He also co-chairs the European 3D Media Cluster, which serves as a contact gateway for information exchange between the associated European projects and international 3D media activities.
His research interests include motion and disparity estimation, 3D media representation and coding, 3D graphics-based scene reconstruction with multi-texture surfaces, and 3D metadata and content description. He has been actively involved in MPEG activities, standardizing the multi-view description for MPEG-7, the view-dependent multi-texturing methods for the 3D scene representation in MPEG4-AFX and contributing to the multi-view video coding process in MPEG4-MVC and 3D Video. In recent projects he was involved in research and development of traffic surveillance systems and visualization of multiple-view video, 3D scene reconstruction, object segmentation, tracking and 3D reconstruction, 3D scene and object representation and interactive user navigation in 3D environments. Currently, he is a Project Manager for European projects in the field of 3D video technology and multimedia content description. He is senior member of the IEEE.

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Klagenfurt University. He joined the Klagenfurt University in 1999 and is currently a Assistant Professor (Ass.-Prof.) at the Department of Information Technology (ITEC) – Multimedia Communication Group. His research interests include the transport of multimedia content, multimedia adaptation in constrained and streaming environments, distributed multimedia adaptation, and QoS/QoE. He has published more than 50 papers (incl. book chapters and tutorials) in these areas and the general chair of WIAMIS2008. Finally, he is an editorial board member of the Encyclopedia of Multimedia, ACM/Springer International Journal on Multimedia Tools and Applications (MTAP), and associate editor for IEEE Computer Science Computing Now.
He has been actively participating in several EC-funded projects, notably the FP6-IST-DANAE (2004-2006), FP6-IST-ENTHRONE (2006-2008), FP7-ICT-P2P-Next (2008-2012), and FP7-ICT-ALICANTE (2010-2013) projects. Dr. Timmerer participated in the work of ISO/MPEG for several years, notably as the head of the Austrian delegation, coordinator of several core experiments, co-chair of several ad-hoc groups, and as an editor for Parts 7 and 8 of MPEG-21, Digital Item Adaptation and Reference Software for which he received ISO/IEC certificates. His current contributions are in the area of the MPEG Extensible Middleware (MXM), MPEG-V (Media Context and Control), Advanced IPTV Terminal (AIT), and MPEG Media Transport (MMT) for which (i.e., MXM and MPEG-V) he also serves as an editor. Publications and MPEG contributions can be found under http://research.timmerer.com.