A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

 A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

IEEE Communications Surveys and Tutorials

[PDF]

Jeroen van der Hooft (Ghent University, Belgium), Hadi Amirpour (AAU, Austria), Maria Torres Vega (KU Leuven, Belgium), Yago Sanchez (Fraunhofer/HHI), Raimund Schatz (AIT, Austria), Thomas Schierl (Fraunhofer/HHI, Germany), and Christian Timmerer (AAU, Austria)

Abstract: Video services are evolving from traditional two-dimensional video to virtual reality and holograms, which offer six degrees of freedom to users, enabling them to freely move around in a scene and change focus as desired. However, this increase in freedom translates into stringent requirements in terms of ultra-high bandwidth (in the order of Gigabits per second) and minimal latency (in the order of milliseconds). To realize such immersive services, the network transport, as well as the video representation and encoding, have to be fundamentally enhanced. The purpose of this tutorial article is to provide an elaborate introduction to the creation, streaming, and evaluation of immersive video. Moreover, it aims to provide lessons learned and to point at promising research paths to enable truly interactive immersive video applications toward holography.

Keywords—Immersive video delivery, 3DoF, 6DoF, omnidirectional video, volumetric video, point clouds, meshes, light fields, holography, end-to-end systems

J. van der Hooft, H. Amirpour, M. Torres Vega, Y. Sanchez, R. Schatz, T. Schierl, C. Timmerer, "A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography," in IEEE Communications Surveys & Tutorials, vol. 25, no. 2, pp. 1336-1375, Secondquarter 2023, doi: 10.1109/COMST.2023.3263252.

Immersive Video Delivery: From Omnidirectional Video to Holography from Alpen-Adria-Universität

MPEG news: a report from the 129th meeting, Brussels, Belgium

The original blog post can be found at the Bitmovin Techblog and has been modified/updated here to focus on and highlight research aspects. Additionally, this version of the blog post will be also posted at ACM SIGMM Records.

MPEG News Archive

The 129th MPEG meeting concluded on January 17, 2020 in Brussels, Belgium with the following topics:

• Coded representation of immersive media – WG11 promotes Network-Based Media Processing (NBMP) to the final stage
• Coded representation of immersive media – Publication of the Technical Report on Architectures for Immersive Media
• Genomic information representation – WG11 receives answers to the joint call for proposals on genomic annotations in conjunction with ISO TC 276/WG 5
• Open font format – WG11 promotes Amendment of Open Font Format to the final stage
• High efficiency coding and media delivery in heterogeneous environments – WG11 progresses Baseline Profile for MPEG-H 3D Audio
• Multimedia content description interface – Conformance and Reference Software for Compact Descriptors for Video Analysis promoted to the final stage

Additional Important Activities at the 129th WG 11 (MPEG) meeting

The 129th WG 11 (MPEG) meeting was attended by more than 500 experts from 25 countries working on important activities including (i) a scene description for MPEG media, (ii) the integration of Video-based Point Cloud Compression (V-PCC) and Immersive Video (MIV), (iii) Video Coding for Machines (VCM), and (iv) a draft call for proposals for MPEG-I Audio among others.

The corresponding press release of the 129th MPEG meeting can be found here: https://mpeg.chiariglione.org/meetings/129. This report focused on network-based media processing (NBMP), architectures of immersive media, compact descriptors for video analysis (CDVA), and an update about adaptive streaming formats (i.e., DASH and CMAF).

MPEG picture at friday plenary; © Rob Koenen (Tiledmedia).

Coded representation of immersive media – WG11 promotes Network-Based Media Processing (NBMP) to the final stage

At its 129th meeting, MPEG promoted ISO/IEC 23090-8, Network-Based Media Processing (NBMP), to Final Draft International Standard (FDIS). The FDIS stage is the final vote before a document is officially adopted as an International Standard (IS). During the FDIS vote, publications and national bodies are only allowed to place a Yes/No vote and are no longer able to make any technical changes. However, project editors are able to fix typos and make other necessary editorial improvements.

What is NBMP? The NBMP standard defines a framework that allows content and service providers to describe, deploy, and control media processing for their content in the cloud by using libraries of pre-built 3rd party functions. The framework includes an abstraction layer to be deployed on top of existing commercial cloud platforms and is designed to be able to be integrated with 5G core and edge computing. The NBMP workflow manager is another essential part of the framework enabling the composition of multiple media processing tasks to process incoming media and metadata from a media source and to produce processed media streams and metadata that are ready for distribution to media sinks.

Why NBMP? With the increasing complexity and sophistication of media services and the incurred media processing, offloading complex media processing operations to the cloud/network is becoming critically important in order to keep receiver hardware simple and power consumption low.

Research aspects: NBMP reminds me a bit about what has been done in the past in MPEG-21, specifically Digital Item Adaptation (DIA) and Digital Item Processing (DIP). The main difference is that MPEG now targets APIs rather than pure metadata formats, which is a step forward in the right direction as APIs can be implemented and used right away. NBMP will be particularly interesting in the context of new networking approaches including, but not limited to, software-defined networking (SDN), information-centric networking (ICN), mobile edge computing (MEC), fog computing, and related aspects in the context of 5G.

Coded representation of immersive media – Publication of the Technical Report on Architectures for Immersive Media

At its 129th meeting, WG11 (MPEG) published an updated version of its technical report on architectures for immersive media. This technical report, which is the first part of the ISO/IEC 23090 (MPEG-I) suite of standards, introduces the different phases of MPEG-I standardization and gives an overview of the parts of the MPEG-I suite. It also documents use cases and defines architectural views on the compression and coded representation of elements of immersive experiences. Furthermore, it describes the coded representation of immersive media and the delivery of a full, individualized immersive media experience. MPEG-I enables scalable and efficient individual delivery as well as mass distribution while adjusting to the rendering capabilities of consumption devices. Finally, this technical report breaks down the elements that contribute to a fully immersive media experience and assigns quality requirements as well as quality and design objectives for those elements.

Research aspects: This technical report provides a kind of reference architecture for immersive media, which may help identify research areas and research questions to be addressed in this context.

Multimedia content description interface – Conformance and Reference Software for Compact Descriptors for Video Analysis promoted to the final stage

Managing and organizing the quickly increasing volume of video content is a challenge for many industry sectors, such as media and entertainment or surveillance. One example task is scalable instance search, i.e., finding content containing a specific object instance or location in a very large video database. This requires video descriptors that can be efficiently extracted, stored, and matched. Standardization enables extracting interoperable descriptors on different devices and using software from different providers so that only the compact descriptors instead of the much larger source videos can be exchanged for matching or querying. ISO/IEC 15938-15:2019 – the MPEG Compact Descriptors for Video Analysis (CDVA) standard – defines such descriptors. CDVA includes highly efficient descriptor components using features resulting from a Deep Neural Network (DNN) and uses predictive coding over video segments. The standard is being adopted by the industry. At its 129th meeting, WG11 (MPEG) has finalized the conformance guidelines and reference software. The software provides the functionality to extract, match, and index CDVA descriptors. For easy deployment, the reference software is also provided as Docker containers.

Research aspects: The availability of reference software helps to conduct reproducible research (i.e., reference software is typically publicly available for free) and the Docker container even further contributes to this aspect.

DASH and CMAF

The 4th edition of DASH has already been published and is available as ISO/IEC 23009-1:2019. Similar to previous iterations, MPEG’s goal was to make the newest edition of DASH publicly available for free, with the goal of industry-wide adoption and adaptation. During the most recent MPEG meeting, we worked towards implementing the first amendment which will include additional (i) CMAF support and (ii) event processing models with minor updates; these amendments are currently in draft and will be finalized at the 130th MPEG meeting in Alpbach, Austria. An overview of all DASH standards and updates are depicted in the figure below:

ISO/IEC 23009-8 or “session-based DASH operations” is the newest variation of MPEG-DASH. The goal of this part of DASH is to allow customization during certain times of a DASH session while maintaining the underlying media presentation description (MPD) for all other sessions. Thus, MPDs should be cacheable within content distribution networks (CDNs) while additional information should be customizable on a per session basis within a newly added session-based description (SBD). It is understood that the SBD should have an efficient representation to avoid file size issues and it should not duplicate information typically found in the MPD.

The 2nd edition of the CMAF standard (ISO/IEC 23000-19) will be available soon (currently under FDIS ballot) and MPEG is currently reviewing additional tools in the so-called ‘technologies under considerations’ document. Therefore, amendments were drafted for additional HEVC media profiles and exploration activities on the storage and archiving of CMAF contents.

The next meeting will bring MPEG back to Austria (for the 4th time) and will be hosted in Alpbach, Tyrol. For more information about the upcoming 130th MPEG meeting click here.

Click here for more information about MPEG meetings and their developments

Workshop on Coding Technologies for Immersive Audio/Visual Experiences

This is an invitation to attend the Workshop on Coding Technologies for Immersive Audio/Visual Experiences. It is open not limited to MPEG members but publicly accessible with free-of-charge preregistration. Non-MPEG member may send your registration information to Lu Yu  and Silke Kenzler  with your name, affiliation and email address.

This workshop will cover the MPEG-I immersive Audio and Visual activities - past, present and future, demonstrate systems and technologies and discuss future requirements on standardization to provide audio/visual immersive experiences.

Time/Date: 13:00-18:00, 10 July, 2019

Address: 

Room: Drottingporten 3

Clarion Post Hotel

Drottningtorget 10

411 03 Gothenburg, Sweden

Programm: 

1300-1315	Introduction  (Lu Yu, Zhejiang University)
1315-1345	Usecases and challenges about user immersive experiences (Valerie Allie, InterDigital)
1345-1415	Overview of technologies for immersive visual experiences: capture, processing, compression, standardization and display (Marek Domanski, Poznan University of Technology)
1415-1445	MPEG-I Immersive Audio (Schuyler Quackenbush, Audio Research Labs)
1445-1455	Brief introduction about demos:      Integral photography display (NHK)     Realtime interactive demo with 3DoF+ content (InterDigital)     Plenoptic 2.0 video camera (Tsinghua University)     A simple free-viewpoint television system (Poznan University of Technology)
1455-1530	Demos  Coffee break
1530-1600	360° and 3DoF+ video (Bart Kroon, Philips)
1600-1630	Point cloud compression (Marius Preda, Telecom SudParis, CNRS Samovar)
1630-1700	How can we achieve 6DoF video compression? (Joel Jung, Orange)
1700-1730	How can we achieve lenslet video compression?  (Xin Jin, Tsinghua University,  Mehrdad Teratani, Nagoya University)
1730-1800	Discussion

QoMEX'17 Review: Down the Rabbit Hole - Immersive Experience

During QoMEX 2017 in Erfurt, Germany we had a special session entitled "Down the Rabbit Hole" which I have introduced here already. The papers of the special session will appear soon in IEEEXplore but together with my co-organizers of this special session -- Raimund Schatz and Judith Redi -- we also wanted to run the special session in a special way. Therefore, we asked authors to prepare concise and thought-provoking paper presentations (~15min incl. Q&A -- paper title, presenter, picture, key words below) to save some time for a panel discussion. Surprisingly, it worked very well and the special session turned out to be worthwhile and informative. In order to keep the audience connected and involved we posted a single slide of all panelists (i.e., paper presenters) which was shown all the time (see below).

The discussion was centered around the question "what is your understanding of a fully immersive experience" which revealed interesting aspects and finally resulted in the main challenge how to quantify immersive experience. In this context, Mr. T. (only those who've been at QoMEX and in this session know why he is called Mr. T. -- join us next time and we will explain you what's behind) raised an interesting idea to interpret the Turing test for immersive experience. That is, fully or truly immersive experience is achieved if a human is no longer aware that she/he actually interacts with cyber-physical systems. I think this statement sets the bar (high) but definitely worth to consider.

Finally, I'd like to thank all presenters/panelists for an amazing special session at QoMEX'17 but the journey is not yet over. I'll be attending ACM MMSys and IEEE ICME presenting/discussing various aspects of immersive experiences; also at the MPEG meeting in Torino which will be dedicated to standardization aspects of immersive experiences.

Also big big thanks to the conference organizers, the team around the general chair Alexander Raake
(TU Ilmenau, Germany), for hosting such a wonderful event! Hope seeing you all next year for QoMEX 2018.

Feel free to test/play around with Bitmovin solutions for VR/360-degree streaming and if you have a RICOH THETA S check out my blog post how to setup a live streaming session.

Come and join us on the journey down the rabbit hole which eventually will lead to wonderland.

MPEG news: a report from the 118th meeting, Hobart, Australia

The original blog post can be found at the Bitmovin Techblog and has been updated here to focus on and highlight research aspects. Additionally, this version of the blog post will be also posted at ACM SIGMM Records.

MPEG News Archive

The entire MPEG press release can be found here comprising the following topics:

• Coded Representation of Immersive Media (MPEG-I): new work item approved and call for test data issued
• Common Media Application Format (CMAF): FDIS approved
• Beyond High Efficiency Video Coding (HEVC): call for evidence for "beyond HEVC" and verification tests for screen content coding extensions of HEVC

Coded Representation of Immersive Media (MPEG-I)

MPEG started to work on the new work item referred to as ISO/IEC 23090 with the "nickname" MPEG-I targeting future immersive applications. The goal of this new standard is to enable various forms of audio-visual immersion including panoramic video with 2D and 3D audio with various degrees of true 3D visual perception. It currently comprises five parts: (pt. 1) a technical report describing the scope of this new standard and a set of use cases and applications; (pt. 2) an application format for omnidirectional media (aka OMAF) to address the urgent need of the industry for a standard is this area; (pt. 3) immersive video which is a kind of placeholder for the successor of HEVC (if at all); (pt. 4) immersive audio as a placeholder for the successor of 3D audio (if at all); and (pt. 5) for point cloud compression. The point cloud compression standard targets lossy compression for point clouds in real-time communication, six Degrees of Freedom (6 DoF) virtual reality, and the dynamic mapping for autonomous driving, cultural heritage applications, etc. Part 2 is related to OMAF which I've discussed in my previous blog post.

MPEG also established an Ad-hoc Group (AhG) on immersive Media quality evaluation with the following mandates: 1. Produce a document on VR QoE requirements; 2. Collect test material with immersive video and audio signals; 3. Study existing methods to assess human perception and reaction to VR stimuli; 4. Develop test methodology for immersive media, including simultaneous video and audio; 5. Study VR experience metrics and their measurability in VR services and devices. AhGs are open to everybody and mostly discussed using mailing lists (join here https://lists.aau.at/mailman/listinfo/immersive-quality). Interestingly, a Joint Qualinet-VQEG team on Immersive Media (JQVIM) has been recently established with similar goals and also the VR Industry Forum (VRIF) has issued a call for VR360 content. It seems there's a strong need for a dataset similar to the one we have created for MPEG-DASH long time ago.

The JQVIM has been created as part of the QUALINET task force on "Immersive Media Experiences (IMEx)" which aims at providing end users the sensation of being part of the particular media which shall result in a worthwhile, informative user and quality of experience. The main goals are providing datasets and tools (hardware/software), subjective quality evaluations, field studies, cross- validation including a strong theoretical foundation relevant along the empirical databases and tools which hopefully results in a framework, methodology, and best practices for immersive media experiences.

Common Media Application Format (CMAF)

The Final Draft International Standard (FDIS) has been issued at the 118th MPEG meeting which concludes the formal technical development process of the standard. At this point in time national bodies can only vote Yes|No and editorial changes are allowed (if any) before the International Standard (IS) becomes available. The goal of CMAF is to define a single format for the transport and storage of segmented media including audio/video formats, subtitles, and encryption -- it is derived from the ISO Base Media File Format (ISOBMFF). As it's a combination of various MPEG standard it's referred to as an Application Format (AS) which mainly takes existing formats/standards and glues them together for a specific target application. The CMAF standard clearly targets dynamic adaptive streaming (over -- but not limited to -- HTTP) but focusing on the media format only and excluding the manifest format. Thus, the CMAF standard shall be compatible with other formats such as MPEG-DASH and HLS. In fact, HLS has been extended already some time ago to support 'fragmented MP4' which we have demonstrated also and it has been interpreted as a first step towards the harmonization of MPEG-DASH and HLS; at least on the segment format. The delivery of CMAF contents with DASH will be described in part 7 of MPEG-DASH that basically comprises a mapping of CMAF concepts to DASH terms.

From a research perspective, it would be interesting to explore how certain CMAF concepts are able to address current industry needs, specifically in the context of low-latency streaming which has been demonstrated recently.

Beyond HEVC...

The preliminary call for evidence (CfE) on video compression with capability beyond HEVC has been issued and is addressed to interested parties that have technology providing better compression capability than the existing standard, either for conventional video material, or for other domains such as HDR/WCG or 360-degree ("VR") video. Test cases are defined for SDR, HDR, and 360-degree content. This call has been made jointly by ISO/IEC MPEG and ITU-T SG16/Q6 (VCEG). The evaluation of the responses is scheduled for July 2017 and depending on the outcome of the CfE, the parent bodies of the Joint Video Exploration Team (JVET) of MPEG and VCEG collaboration intend to issue a Draft Call for Proposals by the end of the July meeting.

Finally, verification tests have been conducted for the Screen Content Coding (SCC) extensions to HEVC showing exceptional performance. Screen content is video containing a significant proportion of rendered (moving or static) graphics, text, or animation rather than, or in addition to, camera-captured video scenes. For scenes containing a substantial amount of text and graphics, the tests showed a major benefit in compression capability for the new extensions over both the Advanced Video Coding standard and the previous version of the newer HEVC standard without the new SCC features.

The question whether and how new codecs like (beyond) HEVC competes with AV1 is subject to research and development. It has been discussed also in the scientific literature but lacks of vendor neutral comparison which is difficult to achieve and not to compare apples with oranges (due to the high number of different coding tools and parameters). An important aspect which always needs to be considered is one typically compares specific implementations of a coding format and not the standard as the encoding is usually not defined, only the bitstream syntax that implicitly defines the decoder.

Publicly available documents from the 118th MPEG meeting can be found here (scroll down to the end of the page). The next MPEG meeting will be held in Torino, Italy, July 17-21, 2017. Feel free to contact us for any questions or comments.

Joint QUALINET-VQEG team on Immersive Media (JQVIM)

QUALINET and VQEG have an ambition to increase the collaboration between the organizations. As a pilot effort we are hereby proposing a Joint Qualinet-VQEG team on Immersive Media (JQVIM). The actual collaboration will be between the Task Force: "Immersive Media Experiences (IMEx)" of Qualinet and the Working Group "Immersive Media Group (IMG)" of VQEG.

The initial goals for JQVIM are:

• Collecting and producing open source immersive media content and data set
• Establishing and recommending best practices and guidelines
• Collecting and producing open source immersive media tools
• Survey of standardisation activities

Anybody who's interested joining this effort, please contact me. Interestingly, this effort is related to my previous post where the VR Industry Forum (VRIF) calls for VR360 Content and also MPEG established an Ad-Hoc Group (AhG) on Immersive Media Quality Evaluation with similar mandates.

Top 10 Blog Posts

1. HTTP Streaming of MPEG Media: My first article in this series which I've started after the MPEG CfP has been issued that lead to the standardization of DASH.
2. MMSys'11 Special Session on MMT/DASH: the CfP for a special session I've organized.
3. MPEG news: a report from the 93rd meeting in Geneva, right after the responses to the HTTP streaming CfP has been evaluated.
4. MPEG advances DASH towards completion which is the MPEG press release after the 94th meeting in Guangzhou.
5. Open Source Scalable Video Coding (SVC) Software where I have received quite a few comments ;-)
6. MPEG Media Transport: Basically the same as #1 but a different scope. However, it seems the readers are more interested in HTTP streaming than media transport in general.
7. Vision and Requirements for High-Performance Video Coding which has been renamed now to High-Efficiency Video Coding.
8. DASH provides an overview about the Draft International Standard which is publicly available.
9. MPEG DASH vs. W3C WebTV which is still a hop topic and worth following on both sides...
10. Immersive Future Media Technologies: From 3D Video to Sensory Experience: I'm happy having this one in my top ten. It's the summary of a tutorial I had at ACM Multimedia 2010 together with Karsten Müller.

In general, most of the readers are very much interested in HTTP streaming / DASH / MMT followed by video coding (SVC/HEVC/3DVC) and the Sensory Experience stuff I've started some time ago.

Thanks again for visiting my blog and don't hesitate to leave a comment here and there. I'd love to read your thoughts and feedback.

3rd International Conference on Immersive Telecommunications

Deadline extension for Paper Submission: March 15, 2011
27-29 June 2011, Istanbul, Turkey

http://www.immerscom.org/2011/
[PDF][DOCX]

SCOPE AND GOAL OF THE CONFERENCE

Immersive telecommunications is an emerging multi-disciplinary field with contributions from telecommunications and networking, signal processing, computer vision, virtual reality, spatial audio processing, user interfaces, and human factors. IMMERSCOM aims to bring researchers and scientists from academia and industry together to advance the state of the art in immersive telecommunications sciences, technologies and systems.

TOPICS

The conference solicits original technical papers that are not previously published or currently under review for publication elsewhere. General themes of the conference include but are not limited to:

• Scene Capture for Immersive Communication: Multi-camera and depth sensor based 3D video capture systems, multimodal scene capture, 3D reconstruction and other representations from capture data, view synthesis and interpolation, pervasive sensing
• Communications and Networking for Immersive Applications: Communication for rich 3D visualization, tele-presence, and mixed/augmented reality; QoS for immersive applications; robustness to loss over networks, low latency communication, real-time protocols, synchronization
• Computer Vision for Immersive Communication: 3D reconstruction and modeling, activity detection, face and pose detection and recognition, eye contact, gesture recognition, body tracking, mobility issues, tele-reality, data fusion
• User Interfaces and Human Factors for Immersive Experiences: Stereoscopic, auto-stereoscopic and multi-view displays and interfaces; multi-projector and projector-camera systems;mixed/augmented reality interfaces, interactive environment interfaces, affective user interfaces, haptics; handheld and wearable device interfaces, multimodal interfaces, tele-presence interfaces; perceptual interfaces, human factors and usability
• AudioProcessing for Immersive Communication: Multi-channel echo cancellation, de-reverberation, spatial audio capture and synthesis, microphone and speaker arrays, source localization and diarization, multimodal processing
• Multimedia Coding for Immersive Communication: Coding of audio, video, multi-view video, and depth with various 3D representations; coding for distributed multimedia sensors (camera & microphone) networks, compressed sensing, real-time coding
• Applications, Systems, and Architectures: Tele-presence; gaming; medical and bio-medical applications; educational systems; virtual museums, meetings and classrooms; social networking

TUTORIAL/POSTER/PANEL/DEMO/EXHIBITION PROPOSALS

Proposals for tutorials/posters/panels/demos/exhibitions related to Immersive Telecommunications technologies are solicited. Those who are interested are welcome to submit proposals to the respective Chairs. Please visit the Submissions page for details.

PUBLICATION

Accepted papers will be published by Springer-Verlag in the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (LNICST) series.The proceedings will be available in both soft and hard copies by Springer.

SUBMISSIONS

Click here for detailed submission instructions.

SPECIAL ISSUE

Selected papers from the IMMERSCOM Technical program will be invited for publication in a Special Issue of the EURASIP Journal Signal Processing: Image Communication.

Important Dates

• Paper Submission Deadline: February 1, 2011 March 15, 2011 (extended)
• Notification of Acceptance: March 15, 2011 April 10, 2011 (extended)
• Camera Ready Deadline: April 10, 2011 May 1, 2011 (extended)
• Author Registration Deadline: April 10, 2011 May 1, 2011 (extended)
• Tutorial/Poster/Panel/Demo Proposals: March 1, 2011 April 10, 2011 (extended)
• Early Registration Deadline: June 1, 2011

Organizing Committee

General Co-Chairs: 

• Murat Tekalp, Koc University, Turkey
• Sabine Coquillart, INRIA, France

Technical Program Co-Chairs:

• Marcus Magnor, TU Braunschweig, Germany
• Cha Zhang, Microsoft Research, USA
• Ozgur B. Akan, Koc Univeristy, Turkey

Publication Chair

• Aydin Alatan, METU, Turkey

Tutorials and Panels Chair

• Dinei Florencio, Microsoft Research, USA

Posters and Demo Chair

• Thierry Duval, Universite de Rennes, France
• Kiyoshi Kiyokawa, Osaka University, Japan

Exhibition and Sponsorship Chair

• Khaled El-Maleh, Qualcomm, USA

Web Chair

• Mashhour Solh, Georgia Institute of Technology, USA

Publicity Co-Chairs

• Dzmitry Tsetserukou, Toyohashi University of Technology, Japan
• Christian Timmerer, Klagenfurt University, Austria

Conference Coordinator

• Edit Marosi, ICST, Hungary

Immersive Future Media Technologies: From 3D Video to Sensory Experience

--tutorial to be held at ACM Multimedia 2010, Oct 25th, Morning, Florence, Italy

--download as PDF

For registration details, please consult ACM Multimedia 2010 Web site!

Instructors: Karsten Müller (Fraunhofer/Heinrich-Hertz-Institut, Berlin) and Christian Timmerer (Klagenfurt University, Austria)

Abstract: The past decade has witnessed a significant increase in the research efforts around the Quality of Experience (QoE) which is generally referred to as a human-centric paradigm for the Quality of a Service (QoS) as perceived by the (end) user. As it puts the end user in the center stage, it may have various dimensions and one dimension recently gained momentum is 3D video. Another dimension aims at going beyond 3D and promises advanced user experience through sensory effects, both introduced briefly in the following.

3D Video: Stereo and Multi-View Video Technology
3D related media technologies have recently developed from pure research-oriented work towards applications and products. 3D content is now being produced on a wider scale and first 3D applications have been standardized, such as multi-view video coding for 3D Blu Ray disks. This part of the tutorial starts with an overview on 3D in the form of stereo video based systems, which are currently being commercialized. Here, stereo formats and associated coding are introduced. This technology is used for 3D cinema applications and mobile 3DTV environments. For the latter, user requirements and profiling will be introduced as a form to assess user quality of experience. For 3D home entertainment, glasses-free multi-view displays are required, as more than one user will watch 3D content. For such displays, the current stereo solutions need to be extended. Therefore, new activities in 3D video are introduced. These 3D solutions will develop a generic 3D video format with color and supplementary geometry data, e.g. depth maps, and associated coding and rendering technology for any multi-view display, independent of the number of views. As such technology is also developed in international consortia, the most prominent, like the 3D@HOME consortium, the EU 3D, Immersive, Interactive Media Cluster and the 3D video activities in ISO-MPEG are introduced.

Advanced User Experience through Sensory Effects
This part of the tutorial addresses a novel approach for increasing the user experience – beyond 3D – through sensory effects. The motivation behind this work is that the consumption of multimedia assets may stimulate also other senses than vision or audition, e.g., olfaction, mechanoreception, equilibrioception, or thermoception that shall lead to an enhanced, unique user experience. This could be achieved by annotating the media resources with metadata (currently defined by ISO/MPEG as part of the MPEG-V standard) providing so-called sensory effects that steer appropriate devices capable of rendering these effects (e.g., fans, vibration chairs, ambient lights, perfumer, water sprayers, fog machines, etc.). In particular, we will review the concepts and details of the forthcoming MPEG-V standard and present our prototype architecture for the generation, transport, decoding and use of sensory effects. Furthermore, we will present details and results of a series of formal subjective quality assessments which confirm that the concept of sensory effects is a vital tool for enhancing the user experience.

Course Outline
Introduction and Overview (~10min.)
3D Video: Stereo and Multi-View Video Technology (~90min)

• User Requirements for 3D video technologies
• Stereo Video solutions for 3D cinema and mobile applications
• User experience and profiling for mobile 3DTV
• Format description and coding for stereo and multi-view video technology
• Depth-enhanced 3D video solutions for 3D home entertainment

Advanced User Experience through Sensory Effects (~90min)

• MPEG-V: context and objectives including an overview of all parts.
• In depth review of Part 3 of MPEG-V entitled Sensory Information.
• A test-bed for the quality of multimedia experience evaluation of sensory effects and demonstration.
• How to improve the Quality of Experience through sensory effects? Results from first subjective experiments.
• Conclusions and future work.

Intended Audience
Intermediate, specifically, graduate students and researchers interested in 3D video and sensory experiences.

Biography of Presenters

Karsten Müller received the Dipl.-Ing. and Dr.-Ing. degree from the Technical University of Berlin, Germany, in 1997 and 2006 respectively. In 1993 he spent one year studying Electronics and Communication Engineering at Napier University of Edinburgh/Scotland, including a half-year working period at Integrated Communication Systems Inc. in Westwick near Cambridge/England. In this working period he developed software for voice mail systems and statistical analysis of caller data. In 1996 he joined the Heinrich-Hertz-Institute (HHI) Berlin, Image Processing Department, were he is a project coordinator for European projects in the field of 3D media technology. He also co-chairs the European 3D Media Cluster, which serves as a contact gateway for information exchange between the associated European projects and international 3D media activities.
His research interests include motion and disparity estimation, 3D media representation and coding, 3D graphics-based scene reconstruction with multi-texture surfaces, and 3D metadata and content description. He has been actively involved in MPEG activities, standardizing the multi-view description for MPEG-7, the view-dependent multi-texturing methods for the 3D scene representation in MPEG4-AFX and contributing to the multi-view video coding process in MPEG4-MVC and 3D Video. In recent projects he was involved in research and development of traffic surveillance systems and visualization of multiple-view video, 3D scene reconstruction, object segmentation, tracking and 3D reconstruction, 3D scene and object representation and interactive user navigation in 3D environments. Currently, he is a Project Manager for European projects in the field of 3D video technology and multimedia content description. He is senior member of the IEEE.

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Klagenfurt University. He joined the Klagenfurt University in 1999 and is currently a Assistant Professor (Ass.-Prof.) at the Department of Information Technology (ITEC) – Multimedia Communication Group. His research interests include the transport of multimedia content, multimedia adaptation in constrained and streaming environments, distributed multimedia adaptation, and QoS/QoE. He has published more than 50 papers (incl. book chapters and tutorials) in these areas and the general chair of WIAMIS2008. Finally, he is an editorial board member of the Encyclopedia of Multimedia, ACM/Springer International Journal on Multimedia Tools and Applications (MTAP), and associate editor for IEEE Computer Science Computing Now.
He has been actively participating in several EC-funded projects, notably the FP6-IST-DANAE (2004-2006), FP6-IST-ENTHRONE (2006-2008), FP7-ICT-P2P-Next (2008-2012), and FP7-ICT-ALICANTE (2010-2013) projects. Dr. Timmerer participated in the work of ISO/MPEG for several years, notably as the head of the Austrian delegation, coordinator of several core experiments, co-chair of several ad-hoc groups, and as an editor for Parts 7 and 8 of MPEG-21, Digital Item Adaptation and Reference Software for which he received ISO/IEC certificates. His current contributions are in the area of the MPEG Extensible Middleware (MXM), MPEG-V (Media Context and Control), Advanced IPTV Terminal (AIT), and MPEG Media Transport (MMT) for which (i.e., MXM and MPEG-V) he also serves as an editor. Publications and MPEG contributions can be found under http://research.timmerer.com.

Immersive VR study explores human response to virtual stimuli

--from CS Newsfeed

Researchers in the Presenccia project have conducted several experiments designed to help understand human response to stimuli in immersive virtual environments. One area the studies are helping in is social research—specifically, how individuals would act in situations in which conducting real-world experiments is dangerous, unethical, or impractical. An upcoming virtual experiment, for example, will study how people react to violence in public. The researchers hope to use their findings to develop applications to help people overcome phobias, to create training simulators for combat, flight, and driving, and to aid people use prosthetic limbs. The research team includes members from diverse fields, including the neuroscience, psychology, psychophysics, mechanical engineering, and philosophy communities. (Science Daily)

Interestingly, this is an FP6 Integrated Project mainly comprising universities and no publications in 2008. Seems this project is already over ... mmmh, I thought they could probably contribute to the ongoing MPEG-V activity.