A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

 A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

IEEE Communications Surveys and Tutorials

[PDF]

Jeroen van der Hooft (Ghent University, Belgium), Hadi Amirpour (AAU, Austria), Maria Torres Vega (KU Leuven, Belgium), Yago Sanchez (Fraunhofer/HHI), Raimund Schatz (AIT, Austria), Thomas Schierl (Fraunhofer/HHI, Germany), and Christian Timmerer (AAU, Austria)

Abstract: Video services are evolving from traditional two-dimensional video to virtual reality and holograms, which offer six degrees of freedom to users, enabling them to freely move around in a scene and change focus as desired. However, this increase in freedom translates into stringent requirements in terms of ultra-high bandwidth (in the order of Gigabits per second) and minimal latency (in the order of milliseconds). To realize such immersive services, the network transport, as well as the video representation and encoding, have to be fundamentally enhanced. The purpose of this tutorial article is to provide an elaborate introduction to the creation, streaming, and evaluation of immersive video. Moreover, it aims to provide lessons learned and to point at promising research paths to enable truly interactive immersive video applications toward holography.

Keywords—Immersive video delivery, 3DoF, 6DoF, omnidirectional video, volumetric video, point clouds, meshes, light fields, holography, end-to-end systems

J. van der Hooft, H. Amirpour, M. Torres Vega, Y. Sanchez, R. Schatz, T. Schierl, C. Timmerer, "A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography," in IEEE Communications Surveys & Tutorials, vol. 25, no. 2, pp. 1336-1375, Secondquarter 2023, doi: 10.1109/COMST.2023.3263252.

Immersive Video Delivery: From Omnidirectional Video to Holography from Alpen-Adria-Universität

ACMMM'19 Tutorial: A Journey towards Fully Immersive Media Access

ACM Multimedia 2019

October 21-25, 2019, Nice, France

https://www.acmmm.org/2019/
[PDF]

Date/time: Monday, Oct 21, 2019, 2pm-5:30pm

Lecturers

Christian Timmerer, Alpen-Adria-Universität Klagenfurt & Bitmovin, Inc.
Ali C. Begen, Ozyegin University and Networked Media

Abstract

Universal media access as proposed in the late 90s, early 2000 is now reality. Thus, we can generate, distribute, share, and consume any media content, anywhere, anytime, and with/on any device. A major technical breakthrough was the adaptive streaming over HTTP resulting in the standardization of MPEG-DASH, which is now successfully deployed in HTML5 environments thanks to corresponding media source extensions (MSE). The next big thing in adaptive media streaming is virtual reality applications and, specifically, omnidirectional (360°) media streaming, which is currently built on top of the existing adaptive streaming ecosystems. This tutorial provides a detailed overview of adaptive streaming of both traditional and omnidirectional media within HTML5 environments. The tutorial focuses on the basic principles and paradigms for adaptive streaming – both traditional and omnidirectional media – as well as on already deployed content generation, distribution, and consumption workflows. Additionally, the tutorial provides insights into standards and emerging technologies in the adaptive streaming space. Finally, the tutorial includes the latest approaches for immersive media streaming enabling 6DoF DASH through Point Cloud Compression (PCC) and concludes with open research issues and industry efforts in this domain.

Keywords: Omnidirectional media, HTTP adaptive streaming, over-the-top video, 360° video, virtual reality, immersive media access.

Slides

A Journey Towards Fully Immersive Media Access from Christian Timmerer

Learning Objectives

This tutorial consists of two main parts. In the first part, we provide a detailed overview of the HTML5 standard and show how it can be used for adaptive streaming deployments. In particular, we focus on the HTML5 video, media extensions, and multi-bitrate encoding, encapsulation and encryption workflows, and survey well-established streaming solutions. Furthermore, we present experiences from the existing deployments and the relevant de jure and de facto standards (DASH, HLS, CMAF) in this space. In the second part, we focus on omnidirectional (360) media from creation to consumption as well as first thoughts on dynamic adaptive point cloud streaming. We survey means for the acquisition, projection, coding and packaging of omnidirectional media as well as delivery, decoding and rendering methods. Emerging standards and industry practices are covered as well (OMAF, VR-IF). Both parts present some of the current research trends, open issues that need further exploration and investigation, and various efforts that are underway in the streaming industry.

Upon attending this tutorial, the participants will have an overview and understanding of the following topics:

• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles of omnidirectional (360-degree) media delivery
• Content generation, distribution and consumption workflows for traditional and omnidirectional media
• Standards and emerging technologies in the adaptive streaming space
• Current and future research on traditional and omnidirectional media delivery, specifically enabling 6DoF adaptive streaming through point cloud compression

ACM Multimedia attracts attendees that are quite knowledgeable in specific areas. However, not all are experts across multiple disciplines (such as the subject matter here) and only few are familiar with what is happening in the field and standards. Thus, we believe the proposed tutorial will be of interest to this year’s attendees as much as it did in the past.

Table of Contents

Part I: The HTML5 Standard and Adaptive Streaming

• HTML5 video and media extensions
• Survey of well-established streaming solutions
• Multi-bitrate encoding, and encapsulation and encryption workflows
• The MPEG-DASH standard, Apple HLS and the developing CMAF standard
• Common issues in scaling and improving quality, multi-screen/hybrid delivery

Part II: Omnidirectional (360-degree) Media

• Acquisition, projection, coding and packaging of 360-degree video
• Delivery, decoding and rendering methods
• The developing MPEG-OMAF and MPEG-I standards
• Ongoing industry efforts, specifically towards 6DoF adaptive streaming

Speakers

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Alpen-Adria-Universität (AAU) Klagenfurt. He joined the AAU in 1999 (as a system administrator) and is currently an Associate Professor at the Institute of Information Technology (ITEC) within the Multimedia Communication Group. His research interests include immersive multimedia communication, streaming, adaptation, Quality of Experience, and Sensory Experience. He was the general chair of WIAMIS 2008, QoMEX 2013, MMSys 2016, and PV 2018 and has participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET, and ICoSOLE. He also participated in ISO/MPEG work for several years, notably in the area of MPEG- 21, MPEG-M, MPEG-V, and MPEG-DASH where he also served as standard editor. In 2013, he cofounded Bitmovin (http://www.bitmovin.com/) to provide professional services around MPEG-DASH where he holds the position of the Chief Innovation Officer (CIO) – Head of Research and Standardization. He is a senior member of IEEE and member of ACM, specifically IEEE Computer Society, IEEE Communications Society, and ACM SIGMM. Dr. Timmerer was a guest editor of three special issues for the IEEE Journal on Selected Areas in Communications (JSAC) and currently serves as associate editor for IEEE Transactions on Multimedia. Further information available at http://blog.timmerer.com.

Ali C. Begen is the co-founder of Networked Media, a technology company that offers consulting services to industrial, legal and academic institutions in the IP video space. He has been a research and development engineer since 2001, and has broad experience in mathematical modeling, performance analysis, optimization, standards development, intellectual property and innovation. Between 2007 and 2015, he was with the Video and Content Platforms Research and Advanced Development Group at Cisco, where he designed and developed algorithms, protocols, products and solutions in the service provider and enterprise video domains. Currently, he is also affiliated with Ozyegin University, where he is teaching and advising students in the computer science department. Ali has a PhD in electrical and computer engineering from Georgia Tech. To date, he received a number of academic and industry awards, and was granted 30+ US patents. He held editorial positions in leading magazines and journals, and served in the organizing committee of several international conferences and workshops in the field. He is a senior member of both the IEEE and ACM. In 2016, he was elected distinguished lecturer by the IEEE Communications Society, and in 2018, he was re-elected for another two-year term. More details are at http://ali.begen.net.

A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP

A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP

[PDF] *** open access ***

Abdelhak Bentaleb, Member, IEEE, Bayan Taani, Member, IEEE, Ali C. Begen, Senior Member, IEEE, Christian Timmerer, Senior Member, IEEE, and Roger Zimmermann, Senior Member, IEEE

HAS adaptation scheme classification.

Abstract --- In this survey, we present state-of-the-art bitrate adaptation algorithms for HTTP adaptive streaming (HAS). As a key distinction from other streaming approaches, the bitrate adaptation algorithms in HAS are chiefly executed at each client, i.e., in a distributed manner. The objective of these algorithms is to ensure a high Quality of Experience (QoE) for viewers in the presence of bandwidth fluctuations due to factors like signal strength, network congestion, network reconvergence events, etc. While such fluctuations are common in public Internet, they can also occur in home networks or even managed networks where there is often admission control and QoS tools. Bitrate adaptation algorithms may take factors like bandwidth estimations, playback buffer fullness, device features, viewer preferences, and content features into account, albeit with different weights. Since the viewer’s QoE needs to be determined in real-time during playback, objective metrics are generally used including number of buffer stalls, duration of startup delay, frequency and amount of quality oscillations, and video instability. By design, the standards for HAS do not mandate any particular adaptation algorithm, leaving it to system builders to innovate and implement their own method. This survey provides an overview of the different methods proposed over the last several years.

Citation: A. Bentaleb, B. Taani, A. C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," in IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, Firstquarter 2019.
doi: 10.1109/COMST.2018.2862938

Delivering Traditional and Omnidirectional Media

This tutorial will be given at the following events:

• IEEE ICME 2018, July 23-27, 2018, San Diego, USA

Delivering Traditional and Omnidirectional Media from Christian Timmerer

Abstract

Universal media access as proposed in the late 90s is now closer to reality. Users can generate, distribute and consume almost any media content, anywhere, anytime and with/on any device. A major technical breakthrough was the adaptive streaming over HTTP resulting in the standardization of MPEG-DASH, which is now successfully deployed in most platforms. The next challenge in adaptive media streaming is virtual reality applications and, specifically, omnidirectional (360°) media streaming.

This tutorial first presents a detailed overview of adaptive streaming of both traditional and omnidirectional media, and focuses on the basic principles and paradigms for adaptive streaming. New ways to deliver such media are explored and industry practices are presented. The tutorial then continues with an introduction to the fundamentals of communications over 5G and looks into mobile multimedia applications that are newly enabled or dramatically enhanced by 5G.

A dedicated section in the tutorial covers the much-debated issues related to quality of experience. Additionally, the tutorial provides insights into the standards, open research problems and various efforts that are underway in the streaming industry.

Learning Objectives

Upon attending this tutorial, the participants will have an overview and understanding of the following topics:

• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles of omnidirectional (360) media delivery
• Content generation, distribution and consumption workflows
• Standards and emerging technologies, new delivery schemes in the adaptive streaming space
• Measuring, quantifying and improving quality of experience
• Fundamental technologies of 5G
• Features and services enabled or enhanced by 5G
• Current and future research on delivering traditional and omnidirectional media

Table of Contents

Part I: Streaming (Presented by Dr. Begen and Dr. Timmerer)

• Survey of well-established streaming solutions (DASH, CMAF and Apple HLS)
• HTML5 video and media extensions
• Multi-bitrate encoding, and encapsulation and encryption workflows
• Common issues in scaling and improving quality, multi-screen/hybrid delivery
• Acquisition, projection, coding and packaging of 360 video
• Delivery, decoding and rendering methods
• The developing MPEG-OMAF and MPEG-I standards

Part II: Communications over 5G (Presented by Dr. Ma and Dr. Begen)

• 5G fundamentals: radio access and core network
• Multimedia signal processing and communications
• Emerging mobile multimedia use cases
• Detailed analysis for selected use cases
• Improving QoE

Speakers

Ali C. Begen recently joined the computer science department at Ozyegin University. Previously, he was a research and development engineer at Cisco, where he has architected, designed and developed algorithms, protocols, products and solutions in the service provider and enterprise video domains. Currently, in addition to teaching and research, he provides consulting services to industrial, legal, and academic institutions through Networked Media, a company he co-founded. Begen holds a Ph.D. degree in electrical and computer engineering from Georgia Tech. He received a number of scholarly and industry awards, and he has editorial positions in prestigious magazines and journals in the field. He is a senior member of the IEEE and a senior member of the ACM. In January 2016, he was elected as a distinguished lecturer by the IEEE Communications Society. Further information on his projects, publications, talks, and teaching, standards and professional activities can be foundhttp://ali.begen.net

Liangping Ma is with InterDigital, Inc., San Diego, CA. He is an IEEE Communication Society Distinguished Lecturer focusing on 5G technologies and standards, video communication and cognitive radios. He is an InterDigital delegate to the 3GPP New Radio standards. His current research interests include various aspects about ultra-reliable and low-latency communication, such as channel coding, multiple access and resource allocation. Previously, he led the research on Quality of Experience (QoE) driven system optimization for video streaming and interactive video communication. Prior to joining InterDigital in 2009, he was with San Diego Research Center and Argon ST (acquired by Boeing), where he led research on cognitive radios and wireless sensor networks and served as the principal investigators of two projects supported by the Department of Defense and the National Science Foundation, respectively. He is the co-inventor of more than 40 patents and the author/co-author of more than 50 journal and conference papers. He has been the Chair of the San Diego Chapter of the IEEE Communication Society since 2014. He received his PhD from University of Delaware in 2004 and his B.S. from Wuhan University, China, in 1998.

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constrained environments) both from the Alpen-Adria-Universität (AAU) Klagenfurt. He joined the AAU in 1999 (as a system administrator) and is currently an Associate Professor at the Institute of Information Technology (ITEC) within the Multimedia Communication Group. His research interests include immersive multimedia communications, streaming, adaptation, quality of experience, and sensory experience. He was the general chair of WIAMIS 2008, QoMEX 2013 and MMSys 2016, and has participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET and ICoSOLE. He also participated in ISO/MPEG work for several years, notably in the area of MPEG-21, MPEG-M, MPEG-V, and MPEG-DASH where he also served as a standard editor. In 2012, he co-founded Bitmovin to provide professional services around MPEG-DASH where he currently holds the position of the Chief Innovation Officer (CIO).

Adaptive Streaming of Traditional and Omnidirectional Media

This tutorial will be given at the following events:

• IEEE ICME 2017, JULY 10-14, 2017, HONG KONG: http://www.icme2017.org/tutorials/
• ACM SIGCOMM 2017, August 21-25, 2017, Los Angeles, CA, USA: http://conferences.sigcomm.org/sigcomm/2017/tutorial-adaptive-streaming.html
• ...

Adaptive Streaming of Traditional and Omnidirectional Media from Christian Timmerer

Abstract

This tutorial consists of three main parts. In the first part, we provide a detailed overview of the HTML5 standard and show how it can be used for adaptive streaming deployments. In particular, we focus on the HTML5 video, media extensions, and multi-bitrate encoding, encapsulation and encryption workflows, and survey well-established streaming solutions. Furthermore, we present experiences from the existing deployments and the relevant de jure and de facto standards (DASH, HLS, CMAF) in this space. In the second part, we focus on omnidirectional (360°) media from creation to consumption. We survey means for the acquisition, projection, coding and packaging of omnidirectional media as well as delivery, decoding and rendering methods. Emerging standards and industry practices are covered as well. The last part presents some of the current research trends, open issues that need further exploration and investigation, and various efforts that are underway in the streaming industry.

Target Audience and Prerequisite Knowledge

This tutorial includes both introductory and advanced level information. The audience is expected of understanding of basic video coding and IP networking principles. Researchers, developers, content and service providers are all welcome.

Table of Contents

• Part I: The HTML5 Standard and Adaptive Streaming
• HTML5 video and media extensions
• Survey of well-established streaming solutions
• Multi-bitrate encoding, and encapsulation and encryption workflows
• The MPEG-DASH standard, Apple HLS and the developing CMAF standard
• Part II: Omnidirectional (360°) Media
• Acquisition, projection, coding and packaging of 360° video
• Delivery, decoding and rendering methods
• The developing MPEG-OMAF and MPEG-I standards
• Part III: Open Issues and Future Directions
• Common issues in scaling and improving quality, multi-screen/hybrid delivery
• Ongoing industry efforts

Speakers

Ali C. Begen recently joined the computer science department at Ozyegin University. Previously, he was a research and development engineer at Cisco, where he has architected, designed and developed algorithms, protocols, products and solutions in the service provider and enterprise video domains. Currently, in addition to teaching and research, he provides consulting services to industrial, legal, and academic institutions through Networked Media, a company he co-founded. Begen holds a Ph.D. degree in electrical and computer engineering from Georgia Tech. He received a number of scholarly and industry awards, and he has editorial positions in prestigious magazines and journals in the field. He is a senior member of the IEEE and a senior member of the ACM. In January 2016, he was elected as a distinguished lecturer by the IEEE Communications Society. Further information on his projects, publications, talks, and teaching, standards and professional activities can be found at http://ali.begen.net.

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constrained environments) both from the Alpen-Adria-Universität (AAU) Klagenfurt. He joined the AAU in 1999 (as a system administrator) and is currently an Associate Professor at the Institute of Information Technology (ITEC) within the Multimedia Communication Group. His research interests include immersive multimedia communications, streaming, adaptation, Quality of Experience, and Sensory Experience. He was the general chair of WIAMIS 2008, QoMEX 2013, and MMSys 2016 and has participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET, and ICoSOLE. He also participated in ISO/MPEG work for several years, notably in the area of MPEG-21, MPEG-M, MPEG-V, and MPEG-DASH where he also served as standard editor. In 2012 he cofounded Bitmovin (http://www.bitmovin.com/) to provide professional services around MPEG-DASH where he holds the position of the Chief Innovation Officer (CIO).

ICME 2015: Over-the-Top Content Delivery: State of the Art and Challenges Ahead

Supported by http://www.dash-player.com/

Tutorial at ICME 2015

June 29 - July 3, 2015

Torino, Italy

http://www.icme2015.ieee-icme.org/

Abstract: Over-the-top content delivery is becoming increasingly attractive for both live and on-demand content thanks to the popularity of platforms like YouTube, Vimeo, Netflix, Hulu, Maxdome, etc. In this tutorial, we present state of the art and challenges ahead in over-the-top content delivery. In particular, the goal of this tutorial is to provide an overview of adaptive media delivery, specifically in the context of HTTP adaptive streaming (HAS) including the recently ratified MPEG-DASH standard. The main focus of the tutorial will be on the common problems in HAS deployments such as client design, QoE optimization, multi-screen and hybrid delivery scenarios, and synchronization issues. For each problem, we will examine proposed solutions along with their pros and cons. In the last part of the tutorial, we will look into the open issues and review the work-in-progress and future research directions.

The tutorial will be held on June 29, 2015 in the afternoon.

Slides will be provided on time and a preliminary version (from previous presentations) can be found here and here.

Biography of Presenters

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Alpen-Adria-Universität Klagenfurt. He is currently an Associate Professor at the Institute of Information Technology (ITEC) within the Multimedia Communication Group. His research interests include immersive multimedia communication, streaming, adaptation, Quality of Experience, and Sensory Experience.

He has published more than 150 papers in these areas and he has organized a number of special sessions and issues in this domain, e.g., “Special Session on MMT/DASH” (MMsys 2011, followed by a special issue in Signal Processing: Image Communication, 2012), “Special Issue on Adaptive Media Streaming” (IEEE JSAC, published 2014). Furthermore, he was the general chair of WIAMIS 2008, QoMEX 2013, and QCMan 2014; will be general chair of ACM Multimedia Systems 2016. He is an editorial board member of IEEE Computer, associate editor for IEEE Transactions on Multimedia, area editor for the Elsevier journal on Signal Processing: Image Communication and a key member of the Interest Groups (IG) on Image and Video Coding as well as Quality of Experience and Director of the Review Board of the IEEE Multimedia Communication Technical Committee. Finally, he writes a regular column for ACM SIGMM Records where he serves as an editor and he is a member of the ACM SIGMM Open Source Software Committee. Dr. Timmerer participated in the work of ISO/MPEG for more then 10 years, notably as the head of the Austrian delegation, coordinator of several core experiments, co-chair of several ad- hoc groups, and as an editor for various standards, notably the MPEG-21 Multimedia Framework and the MPEG Extensible Middleware (MXM which became MPEG-M). His current contributions are in the area of MPEG-V (Media Context and Control) and Dynamic Adaptive Streaming over HTTP (DASH), for which he also serves as an editor. He received various ISO/IEC certificates of appreciation.

Ali C. Begen is with the Video and Content Platforms Research and Advanced Development Group at Cisco. His interests include networked entertainment, Internet multimedia, transport protocols and content delivery. Ali is currently working on architectures and protocols for next-generation video transport and distribution over IP networks, and he is an active contributor in the IETF and MPEG in these areas. Ali holds a Ph.D. degree in electrical and computer engineering from Georgia Tech. He received the Best Student-paper Award at IEEE ICIP 2003, the Most-cited Paper Award from Elsevier Signal Processing: Image Communication in 2008, and the Best-paper Award at Packet Video Workshop 2012. Ali has been an editor for the Consumer Communications and Networking series in the IEEE Communications Magazine since 2011 and an associate editor for the IEEE Transactions on Multimedia since 2013. He is a senior member of the IEEE and a senior member of the ACM. Further information on Ali’s projects, publications and presentations can be found at http://ali.begen.net.

Over-the-Top Content Delivery: State of the Art and Challenges Ahead

Tutorial at ACM Multimedia 2014

November 3-7, 2014

Orlando, Florida, USA

http://acmmm.org/2014/

Abstract: In this tutorial we present state of the art and challenges ahead in over-the-top content delivery. It particular, the goal of this tutorial is to provide an overview of adaptive media delivery, specifically in the context of HTTP adaptive streaming (HAS) including the recently ratified MPEG-DASH standard. The main focus of the tutorial will be on the common problems in HAS deployments such as client design, QoE optimization, multi-screen and hybrid delivery scenarios, and synchronization issues. For each problem, we will examine proposed solutions along with their pros and cons. In the last part of the tutorial, we will look into the open issues and review the work-in-progress and future research directions.

Slides:

Over the Top Content Delivery: State of the Art and Challenges Ahead from Christian Timmerer

Biography of Presenters

Christian Timmerer is an Associate Professor at the Institute of Information Technology (ITEC), Multimedia Communication Group (MMC), Alpen-Adria-Universität Klagenfurt, Austria. His research interests include immersive multimedia communication, streaming, adaptation, and Quality of Experience (QoE). He was the general chair of QoMEX’13, WIAMIS’08, AVSTP2P’10 (co-located with ACMMM’10), WoMAN’11 (co-located with ICME’11), and TPC co-chair of QoMEX’12. He has participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, and ICoSOLE. He is an Associate Editor for IEEE Computer Science Computing Now, Area Editor for Elsevier Signal Processing: Image Communication, Review Board Member of IEEE MMTC, editor of ACM SIGMM Records, and member of ACM SIGMM Open Source Software Committee. He also participated in ISO/MPEG work for several years, notably in the area of MPEG-21, MPEG-M, MPEG-V, and DASH (incl. DASH Industry Forum). He received his PhD in 2006 from the Klagenfurt University. Follow him on http://www.twitter.com/timse7 and subscribe to his blog http://blog.timmerer.com. 

Ali C. Begen is with the Video and Content Platforms Research and Advanced Development Group at Cisco. His interests include networked entertainment, Internet multimedia, transport protocols and content delivery. Ali is currently working on architectures and protocols for next-generation video transport and distribution over IP networks. He is an active contributor in the IETF and MPEG, and has given a number of keynotes, tutorials and guest lectures in these areas. Ali holds a Ph.D. degree in electrical and computer engineering from Georgia Tech. He received the Best Student-paper Award at IEEE ICIP 2003, the Most-cited Paper Award from Elsevier Signal Processing: Image Communication in 2008, and the Best-paper Award at Packet Video Workshop 2012. Ali has been an editor for the Consumer Communications and Networking series in the IEEE Communications Magazine since 2011 and an associate editor for the IEEE Transactions on Multimedia since 2013. He served as a general co-chair for ACM Multimedia Systems 2011 and Packet Video Workshop 2013. He is a senior member of the IEEE and a senior member of the ACM. Further information on Ali’s projects, publications, presentations and professional activities can be found at http://ali.begen.net.

MPEG-DASH Tutorials

Update: Feb 26, 2022

I realized that this blog post is used quite a lot but heavily outdated. Thus, I decided to post here the most recent DASH tutorial.

Delivering Traditional and Omnidirectional Media from Christian Timmerer

Please note that another one is coming up during ACM MM 2019 in October 2019 (slides below).

A Journey Towards Fully Immersive Media Access from Christian Timmerer

Adaptive Streaming of Traditional and Omnidirectional Media from Christian Timmerer

Over the Top Content Delivery: State of the Art and Challenges Ahead from Christian Timmerer

Immersive Future Media Technologies: From 3D Video to Sensory Experience

--tutorial to be held at ACM Multimedia 2010, Oct 25th, Morning, Florence, Italy

--download as PDF

For registration details, please consult ACM Multimedia 2010 Web site!

Instructors: Karsten Müller (Fraunhofer/Heinrich-Hertz-Institut, Berlin) and Christian Timmerer (Klagenfurt University, Austria)

Abstract: The past decade has witnessed a significant increase in the research efforts around the Quality of Experience (QoE) which is generally referred to as a human-centric paradigm for the Quality of a Service (QoS) as perceived by the (end) user. As it puts the end user in the center stage, it may have various dimensions and one dimension recently gained momentum is 3D video. Another dimension aims at going beyond 3D and promises advanced user experience through sensory effects, both introduced briefly in the following.

3D Video: Stereo and Multi-View Video Technology
3D related media technologies have recently developed from pure research-oriented work towards applications and products. 3D content is now being produced on a wider scale and first 3D applications have been standardized, such as multi-view video coding for 3D Blu Ray disks. This part of the tutorial starts with an overview on 3D in the form of stereo video based systems, which are currently being commercialized. Here, stereo formats and associated coding are introduced. This technology is used for 3D cinema applications and mobile 3DTV environments. For the latter, user requirements and profiling will be introduced as a form to assess user quality of experience. For 3D home entertainment, glasses-free multi-view displays are required, as more than one user will watch 3D content. For such displays, the current stereo solutions need to be extended. Therefore, new activities in 3D video are introduced. These 3D solutions will develop a generic 3D video format with color and supplementary geometry data, e.g. depth maps, and associated coding and rendering technology for any multi-view display, independent of the number of views. As such technology is also developed in international consortia, the most prominent, like the 3D@HOME consortium, the EU 3D, Immersive, Interactive Media Cluster and the 3D video activities in ISO-MPEG are introduced.

Advanced User Experience through Sensory Effects
This part of the tutorial addresses a novel approach for increasing the user experience – beyond 3D – through sensory effects. The motivation behind this work is that the consumption of multimedia assets may stimulate also other senses than vision or audition, e.g., olfaction, mechanoreception, equilibrioception, or thermoception that shall lead to an enhanced, unique user experience. This could be achieved by annotating the media resources with metadata (currently defined by ISO/MPEG as part of the MPEG-V standard) providing so-called sensory effects that steer appropriate devices capable of rendering these effects (e.g., fans, vibration chairs, ambient lights, perfumer, water sprayers, fog machines, etc.). In particular, we will review the concepts and details of the forthcoming MPEG-V standard and present our prototype architecture for the generation, transport, decoding and use of sensory effects. Furthermore, we will present details and results of a series of formal subjective quality assessments which confirm that the concept of sensory effects is a vital tool for enhancing the user experience.

Course Outline
Introduction and Overview (~10min.)
3D Video: Stereo and Multi-View Video Technology (~90min)

• User Requirements for 3D video technologies
• Stereo Video solutions for 3D cinema and mobile applications
• User experience and profiling for mobile 3DTV
• Format description and coding for stereo and multi-view video technology
• Depth-enhanced 3D video solutions for 3D home entertainment

Advanced User Experience through Sensory Effects (~90min)

• MPEG-V: context and objectives including an overview of all parts.
• In depth review of Part 3 of MPEG-V entitled Sensory Information.
• A test-bed for the quality of multimedia experience evaluation of sensory effects and demonstration.
• How to improve the Quality of Experience through sensory effects? Results from first subjective experiments.
• Conclusions and future work.

Intended Audience
Intermediate, specifically, graduate students and researchers interested in 3D video and sensory experiences.

Biography of Presenters

Karsten Müller received the Dipl.-Ing. and Dr.-Ing. degree from the Technical University of Berlin, Germany, in 1997 and 2006 respectively. In 1993 he spent one year studying Electronics and Communication Engineering at Napier University of Edinburgh/Scotland, including a half-year working period at Integrated Communication Systems Inc. in Westwick near Cambridge/England. In this working period he developed software for voice mail systems and statistical analysis of caller data. In 1996 he joined the Heinrich-Hertz-Institute (HHI) Berlin, Image Processing Department, were he is a project coordinator for European projects in the field of 3D media technology. He also co-chairs the European 3D Media Cluster, which serves as a contact gateway for information exchange between the associated European projects and international 3D media activities.
His research interests include motion and disparity estimation, 3D media representation and coding, 3D graphics-based scene reconstruction with multi-texture surfaces, and 3D metadata and content description. He has been actively involved in MPEG activities, standardizing the multi-view description for MPEG-7, the view-dependent multi-texturing methods for the 3D scene representation in MPEG4-AFX and contributing to the multi-view video coding process in MPEG4-MVC and 3D Video. In recent projects he was involved in research and development of traffic surveillance systems and visualization of multiple-view video, 3D scene reconstruction, object segmentation, tracking and 3D reconstruction, 3D scene and object representation and interactive user navigation in 3D environments. Currently, he is a Project Manager for European projects in the field of 3D video technology and multimedia content description. He is senior member of the IEEE.

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Klagenfurt University. He joined the Klagenfurt University in 1999 and is currently a Assistant Professor (Ass.-Prof.) at the Department of Information Technology (ITEC) – Multimedia Communication Group. His research interests include the transport of multimedia content, multimedia adaptation in constrained and streaming environments, distributed multimedia adaptation, and QoS/QoE. He has published more than 50 papers (incl. book chapters and tutorials) in these areas and the general chair of WIAMIS2008. Finally, he is an editorial board member of the Encyclopedia of Multimedia, ACM/Springer International Journal on Multimedia Tools and Applications (MTAP), and associate editor for IEEE Computer Science Computing Now.
He has been actively participating in several EC-funded projects, notably the FP6-IST-DANAE (2004-2006), FP6-IST-ENTHRONE (2006-2008), FP7-ICT-P2P-Next (2008-2012), and FP7-ICT-ALICANTE (2010-2013) projects. Dr. Timmerer participated in the work of ISO/MPEG for several years, notably as the head of the Austrian delegation, coordinator of several core experiments, co-chair of several ad-hoc groups, and as an editor for Parts 7 and 8 of MPEG-21, Digital Item Adaptation and Reference Software for which he received ISO/IEC certificates. His current contributions are in the area of the MPEG Extensible Middleware (MXM), MPEG-V (Media Context and Control), Advanced IPTV Terminal (AIT), and MPEG Media Transport (MMT) for which (i.e., MXM and MPEG-V) he also serves as an editor. Publications and MPEG contributions can be found under http://research.timmerer.com.