DCC’20: Fast Multi-Rate Encoding for Adaptive HTTP Streaming

Fast Multi-Rate Encoding for Adaptive HTTP Streaming

Data Compression Conference 2020, March 24 – 27, Cliff Lodge, Snowbird, UT

[PDF] (coming soon)

Hadi Amirpour, Ekrem Çetinkaya (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), and Mohammad Ghanbari (University of Tehran, University of Essex)

Abstract: Adaptive HTTP streaming is the preferred method to deliver multimedia content on the internet. It provides multiple representations of the same content in different qualities (i.e., bit-rates and resolutions) and allows the client to request segments from the available representations in a dynamic, adaptive way depending on its context. The growing number of representations in adaptive HTTP streaming makes encoding of one video segment at different representations a challenging task in terms of encoding time-complexity. In this paper, information of both the highest and lowest quality representations are used to limit Rate-Distortion Optimization (RDO) for each Coding Unit Tree (CTU) in High Efficiency Video Coding. Our proposed method first encodes the highest quality representation and consequently uses it to encode the lowest quality representation. In particular, the block structure and the selected reference frame of both the highest and lowest quality representations are then used to predict and shorten the RDO process of each CTU for intermediate quality representations. Our proposed method introduces a delay of two CTUs thanks to employing parallel processing techniques. Experimental results show a significant reduction in time-complexity over the reference software (38%) and state-of-the-art (10%) is achieved while quality degradation is negligible.

Keywords:  HTTP adaptive streaming, Multi-rate encoding, HEVC, Fast block partitioning

MPEG news: a report from the 103rd meeting, Geneva, Switzerland

MPEG plenary meeting at CICG in Geneva, CH

The 103rd MPEG meeting was held in Geneva, Switzerland, January 21-15, 2013. The official press release can be found here (doc only) and I'd like to introduce the new MPEG-H standard (ISO/IEC 23008) referred to as high efficiency coding and media delivery in heterogeneous environments:

• Part 1: MPEG Media Transport (MMT) - status: 2nd committee draft (CD)
• Part 2: High Efficiency Video Coding (HEVC) - status: final draft international standard (FDIS)
• Part 3: 3D Audio - status: call for proposals (CfP)

MPEG Media Transport (MMT)

The MMT project was started in order to address the needs of modern media transport applications going beyond the capabilities offered by existing means of transportation such as formats defined by MPEG-2 transport stream (M2TS) or ISO base media file format (ISOBMFF) group of standards. The committee draft was approved during the 101st MPEG meeting. As a response to the CD ballot, MPEG received more than 200 comments from national bodies and, thus, decided to issue the 2nd committee draft which will be publicly available by February 7, 2013.

High Efficiency Video Coding (HEVC) - ITU-T H.265 | MPEG HEVC

HEVC is the next generation video coding standard jointly developed by ISO/IEC JTC1/SC29/WG11 (MPEG) and the Video Coding Experts Group (VCEG) of ITU-T WP 3/16. Please note that both ITU-T and ISO/IEC MPEG use the term "high efficiency video coding" in the the title of the standard but one can expect - as with its predecessor - that the former will use ITU-T H.265 and the latter will use MPEG-H HEVC for promoting its standards. If you don't want to participate in this debate, simply use high efficiency video coding.

The MPEG press release says that the "HEVC standard reduces by half the bit rate needed to deliver high-quality video for a broad variety of applications" (note: compared to its predecessor AVC). The editing period for the FDIS goes until March 3, 2013 and then with the final preparations and a 2 month balloting period (yes|no vote only) once can expect the International Standard (IS) to be available early summer 2013. Please note that there are no technical differences between FDIS and IS.

The ITU-T press release describes HEVC as a standard that "will provide a flexible, reliable and robust solution, future-proofed to support the next decade of video. The new standard is designed to take account of advancing screen resolutions and is expected to be phased in as high-end products and services outgrow the limits of current network and display technology."

HEVC currently defines three profiles:

• Main Profile for the "Mass-market consumer video products that historically require only 8 bits of precision".
• Main 10 Profile "will support up to 10 bits of processing precision for applications with higher quality demands".
• Main Still Picture Profile to support still image applications, hence, "HEVC also advances the state-of-the-art for still picture coding"

3D Audio

The 3D audio standard shall complement MMT and HEVC assuming that in a "home theater" system a large number of loudspeakers will be deployed. Therefore, MPEG has issued a Call for Proposals (CfP) with the selection of the reference model v0 due in July 2013. The CfP says that MPEG-H 3D Audio "might be surrounding the user and be situated at high, mid and low vertical positions relative to the user’s ears. The desired sense of audio envelopment includes both immersive 3D audio, in the sense of being able to virtualize sound sources at any position in space, and accurate audio localization, in terms of both direction and distance."

"In addition to a “home theater” audio-visual system, there may be a “personal” system having a tablet-sized visual display with speakers built into the device, e.g. around the perimeter of the display. Alternatively, the personal device may be a hand-held smart phone. Headphones with appropriate spatialization would also be a means to deliver an immersive audio experience for all systems."

Complementary to the CfP, MPEG also provided the encoder input format for MPEG-H 3D audio and a draft MPEG audio core experiment methodology for 3D audio work.

Publicly available MPEG output documents

The following documents shall be come available at http://mpeg.chiariglione.org/ (note: some may have an editing period - YY/MM/DD). If you have difficulties to access one of these documents, please feel free to contact me.

• Study text of DIS of ISO/IEC 23000-13, Augmented Reality Application Format (13/01/25)
• Study text of DTR of ISO/IEC 23000-14, Augmented reality reference model (13/02/25)
• Text of ISO/IEC FDIS 23005-1 2nd edition Architecture (13/01/25)
• Text of ISO/IEC 2nd CD 23008-1 MPEG Media Transport (13/02/07)
• Text of ISO/IEC 23008-2:201x/PDAM1 Range Extensions (13/03/22)
• Text of ISO/IEC 23008-2:201x/PDAM2 Multiview Extensions (13/03/22)
• Call for Proposals on 3D Audio (13/01/25)
• Encoder Input Format for MPEG-H 3D Audio (13/02/08)
• Draft MPEG Audio CE methodology for 3D Audio work (13/01/25)
• Draft Requirements on MPEG User Descriptions (13/02/08)
• Draft Call for Proposals on MPEG User Descriptions (13/01/25)
• Draft Call for Proposals on Green MPEG (13/01/25)
• Context, Objectives, Use Cases and Requirements for Green MPEG (13/01/25)
• White Paper on State of the Art in compression and transmission of 3D Video (13/01/28)
• MPEG Awareness Event Flyer at 104th MPEG meeting in Incheon (13/02/28)

MPEG news: a report from the 101st meeting, Stockholm, Sweden

The 101st MPEG meeting was held in Stockholm, Sweden, July 16-20, 2012. The official press release can be found here and I would like to highlight the following topics:

• MPEG Media Transport (MMT) reaches Committee Draft (CD)
• High-Efficiency Video Coding (HEVC) reaches Draft International Standard (DIS)
• MPEG and ITU-T establish JCT-3V
• Call for Proposals: HEVC scalability extensions
• 3D audio workshop
• Green MPEG

MMT goes CD

The Committee Draft (CD) of MPEG-H part 1 referred to as MPEG Media Transport (MMT) has been approved and will be publicly available after an editing period which will end Sep 17th. MMT comprises the following features:

• Delivery of coded media by concurrently using more than one delivery medium (e.g., as it is the case of heterogeneous networks).
• Logical packaging structure and composition information to support multimedia mash-ups (e.g., multiscreen presentation).
• Seamless and easy conversion between storage and delivery formats.
• Cross layer interface to facilitate communication between the application layers and underlying delivery layers.
• Signaling of messages to manage the presentation and optimized delivery of media.

This list of 'features' may sound very high-level but as the CD usually comprises stable technology and is publicly available, the research community is more than welcome to evaluate MPEG's new way of media transport. Having said this, I would like to refer to the Call for Papers of  JSAC's special issue on adaptive media streaming which is mainly focusing on DASH but investigating its relationship to MMT is definitely within the scope.

HEVCs' next step towards completion: DIS

The approval of the Draft International Standard (DIS) brought the HEVC standard one step closer to completion. As reported previously, HEVC shows inferior performance gains compared to its predecessor and real-time software decoding on the iPad 3 (720p, 30Hz, 1.5 Mbps) has been demonstrated during the Friday plenary [1, 2]. It is expected that the Final Draft International Standard (FDIS) is going to be approved at the 103rd MPEG meeting in January 21-25, 2013. If the market need for HEVC is only similar as it was when AVC was finally approved, I am wondering if one can expect first products by mid/end 2013. From a research point of view we know - and history is our witness - that improvements are still possible even if the standard has been approved some time ago. For example, the AVC standard is now available in its 7th edition as a consolidation of various amendments and corrigenda.

JCT-3V

After the Joint Video Team (JVT) which successfully developed standards such as AVC, SVC, MVC and the Joint Collaborative Team on Video Coding (JCT-VC), MPEG and ITU-T establish the Joint Collaborative Team on 3D Video coding extension development (JCT-3V). That is, from now on MPEG and ITU-T also joins forces in developing 3D video coding extensions for existing codecs as well as the ones under development (i.e., AVC, HEVC). The current standardization plan includes the development of AVC multi-view extensions with depth to be completed this year and I assume HEVC will be extended with 3D capabilities once the 2D version is available.

In this context it is interesting that a call for proposals for MPEG Frame Compatible (MFC) has been issued to address current deployment issues of stereoscopic videos. The requirements are available here.

Call for Proposals: SVC for HEVC

In order to address the need for higher resolutions - Ultra HDTV - and subsets thereof, JCT-VC issued a call for proposals for HEVC scalability extensions. Similar to AVC/SVC, the requirements include that the base layer should be compatible with HEVC and enhancement layers may include temporal, spatial, and fidelity scalability. The actual call, the use cases, and the requirements shall become available on the MPEG Web site.

MPEG hosts 3D Audio Workshop

Part 3 of MPEG-H will be dedicated to audio, specifically 3D audio. The call for proposals will be issues at the 102nd MPEG meeting in October 2012 and submissions will be due at the 104th meeting in April 2013. At this meeting, MPEG has hosted a 2nd workshop on 3D audio with the following speakers.

• Frank Melchior, BBC R&D: “3D Audio? - Be inspired by the Audience!”
• Kaoru Watanabe, NHK and ITU: “Advanced multichannel audio activity and requirements”
• Bert Van Daele, Auro Technologies: “3D audio content production, post production and distribution and release”
• Michael Kelly, DTS: “3D audio, objects and interactivity in games”

The report of this workshop including the presentations will be publicly available by end of August at the MPEG Web site.

What's new: Green MPEG

Finally, MPEG is starting to explore a new area which is currently referred to as Green MPEG addressing technologies to enable energy-efficient use of MPEG standards. Therefore, an Ad-hoc Group (AhG) was established with the following mandates:

1. Study the requirements and use-cases for energy efficient use of MPEG technology.
2. Solicit further evidence for the energy savings.
3. Develop reference software for Green MPEG experimentation and upload any such software to the SVN.
4. Survey possible solutions for energy-efficient video processing and presentation.
5. Explore the relationship between metadata types and coding technologies.
6. Identify new metadata that will enable additional power savings.
7. Study system-wide interactions and implications of energy-efficient processing on mobile devices.

AhGs are usually open to the public and all discussions take place via email. To subscribe please feel free to join the email reflector.

MPEG news: a report from the 100th meeting, Geneva, CH

The official press release is available here and I'd like to highlight two topics from MPEGs' 100th meeting in Geneva, Switzerland:

• "MP100E": MPEG celebrates its 100th meeting
• Systems news: ISOBMFF 4th edition, MDS social metadata, DASH conformance/refsw et al.
• WebVC and ARAF goes CD
• HEVC preliminary subjective test results publicly available

MPEG celebrates its 100th meeting

The 100th MPEG meeting was held in Geneva, Switzerland with a sponsored social event including honorific speeches by representatives of ISO, IEC, ITU-T, WIPO, JTC 1, SC 29, Sisvel, Samsung, and MERL. The ISO press release can be found here highlighting the win of three "Technology and Engineering Emmy Awards presented by the U.S. National Academy of Television Arts & Sciences (NATAS) for outstanding achievement in engineering and technical development. The awards covered:

• The MPEG-4 Advanced Video Coding (AVC) standard. The related ITU-T Video Coding Experts Group also received an Emmy Award for its role in the work on this standard
• MPEG-1 and MPEG-2 – compression coding associated with video CD and MP3, digital TV set top boxes and DVD."

Some additional pictures can be found here.

Systems news

ISOBMFF 4th edition: The "Text of ISO/IEC 14496-12 4th edition" has been approved which means the the ISO base media file format is available in its fourth edition (i.e., a consolidation of Amd.1, Amd.2, Amd.3, Cor.1-5). As previous editions, it shall become publicly available via ITTF Web site.

MDS social metadata: The fourth amendment of MPEG-7 MDS has been approved providing support  for social metadata such as ratings (like, 5-star) based on media, identity, and quality.

DASH conference/reference software: The committee draft for ISO/IEC 23009-2 has been approved adding conformance and reference software to DASH. In particular, MPD and segment conformance rules are defined and bitstreams are available. Furthermore, libdash has been selected as sample client. A public online MPD validator is available at http://dash.itec.aau.at.

WebVC and ARAF goes CD

WebVC: The committee draft of MPEG-4 Part 29 aka Web Video Coding (WebVC) has been approved which is compatible with the Constrained Baseline Profile of ISO/IEC 14996-10. In fact, WebVC is derived from the AVC specification with the purpose to define a coding format suitable for the Web. The CD is publicly available here.

ARAF: MPEG addresses the need for standards in the area of augmented realities by defining an application format. Hence, ARAF stands for Augmented Reality Application Format and will be defined as Part 13 of MPEG-A. The committee draft is publicly available here addressing use cases and requirements.

HEVC preliminary subjective test results publicly available

"The video subgroup recommends making the report on preliminary subjective testing N12475 publicly available."

N12475 is the Report on preliminary subjective testing of HEVC compression capability which can be found here. It shows impressive results as reported elsewhere, e.g., here. In particular, > 50% bitrate reduction, 67% in class B (HDTV), 49% in class C (WVGA) => mission accomplished! Currently, HEVC is between ballots and FDIS/IS is expected around Jan-Apr 2013.

MPEG news: a report from the 99th meeting, San Jose, CA, USA

The official press release is available here and I'd like to highlight two topics from MPEGs' 99th meeting in San Jose, CA, USA:

• HEVC advances to Committee Draft (CD)
• Public workshop on MPEG-H 3D Audio

High-Efficiency Video Coding reaches first formal milestone towards completion

As described in the official press release "ISO/IEC’s Moving Picture Experts Group (MPEG) is pleased to announce the completion of the ISO/IEC committee draft of the High Efficiency Video Coding (HEVC) standard developed by the Joint Collaborative Team on Video Coding (JCT-VC), a joint team between MPEG and the ITU-T’s Video Coding Experts Group (VCEG)". For those who are not familiar with the ISO/IEC standardization process, committee draft (CD) means that the standard is not yet finalized but entering the committee stage which enables national bodies to comment on the standard. That is, changes to HEVC can be only made through national body comments which needs to be registered in due time.

In terms of performance of HEVC one can conclude that the mission is accomplished. Preliminary HM5 vs. AVC subjective performance comparison looks impressive, i.e. > 50% bitrate reduction overall, specifically 67% in HD and 49% for WVGA sequences. Please note that these results are not validated through official verification tests which are usually conducted in a later stage of the standardization process.

From a deployment perspective currently one profile is foreseen which is preliminarily referred to as the "main" profile with a largest coding unit (LCU) between 16x16 and 64x64 and a max. pictures storage capacity always 6 (compared to AVC which is max. 16) among others.

Research issues: in my last report I wrote "the ultimative goal to have a performance gain of more than 50% compared to the predecessor which is AVC". It seems this has been achieved so one might wonder what else needs to be done. In practice, however, there is always space for improvement, right?

The next step in audio coding: MPEG-H 3D Audio

The MPEG-H 3D Audio Workshop attracted more than 100 attendees which followed presentations covering three areas of 3D audio.

1. ATSC 3.0 and the Future of Broadcast Television (FoBTV)
2. 22.2 multichannel sound for Ultra High Definition TV (UHDTV), Next Generation Broadcast Television, and New Heights in Multichannel Sound: Explorations and Considerations
3. Realistic audio representation technologies for UHDTV, backward-compatible 3D audio coding, and innovating beyond 5.1.

The presentations are publicly available here within a single ZIP file. MPEG established an AhG on 3D Audio (and Audio Maintenance) with the following mandates (among others):

• Progress possible use cases, requirements and evaluation methods for 3D Audio 
• Identify test material appropriate for 3D Audio work and a process to make the material available to interested MPEG delegates.

Subscription to the reflector is open to everyone. A possible timeline for part 3 of MPEG-H could mean to have a Call for Proposals (CfP) in July 2012 followed by the evaluation in January 2013, all preliminary, no guarantee.

Finally, the next meeting will be MPEGs' 100th meeting which will include a social event with participation of representatives from ITU, ISO, IEC, and others.

Top 10 Blog Posts

1. HTTP Streaming of MPEG Media: My first article in this series which I've started after the MPEG CfP has been issued that lead to the standardization of DASH.
2. MMSys'11 Special Session on MMT/DASH: the CfP for a special session I've organized.
3. MPEG news: a report from the 93rd meeting in Geneva, right after the responses to the HTTP streaming CfP has been evaluated.
4. MPEG advances DASH towards completion which is the MPEG press release after the 94th meeting in Guangzhou.
5. Open Source Scalable Video Coding (SVC) Software where I have received quite a few comments ;-)
6. MPEG Media Transport: Basically the same as #1 but a different scope. However, it seems the readers are more interested in HTTP streaming than media transport in general.
7. Vision and Requirements for High-Performance Video Coding which has been renamed now to High-Efficiency Video Coding.
8. DASH provides an overview about the Draft International Standard which is publicly available.
9. MPEG DASH vs. W3C WebTV which is still a hop topic and worth following on both sides...
10. Immersive Future Media Technologies: From 3D Video to Sensory Experience: I'm happy having this one in my top ten. It's the summary of a tutorial I had at ACM Multimedia 2010 together with Karsten Müller.

In general, most of the readers are very much interested in HTTP streaming / DASH / MMT followed by video coding (SVC/HEVC/3DVC) and the Sensory Experience stuff I've started some time ago.

Thanks again for visiting my blog and don't hesitate to leave a comment here and there. I'd love to read your thoughts and feedback.

Summary of results of the first JCT-VC meeting in Dresden

--text adopted from Gary Sullivan

JCT-VC stands for Joint Collaborative Team on Video Coding and is basically the successors of the Joint Video Team (JVT) which was responsible for standardizing the award-winning Advanced Video Coding (AVC) standard. Last week, the JCT-VC met the first time in Dresden for the evaluation of the joint call for proposals on video compression technology and Gary Sullivan provided a nice summary over the general JCT-VC reflector which is excerpted here.

The name for the new standardization project is now "High Efficiency Video Coding" (HEVC).

The Joint Call for Proposals (CfP) on Video Compression Technology, which was issued by ISO/IEC MPEG and ITU-T VCEG in January 2010, had a very successful outcome. Twenty-seven complete proposal submissions were received, and the associated video material was evaluated in extensive subjective tests that were conducted prior to the first meeting of the Joint Collaborative Team on Video Coding (JCT-VC). That meeting was held under MPEG auspices in Dresden during 15-23 April 2010. The contribution documents for those proposals are at the above link with contribution numbers JCTVC-A101 to JCTVC-A127.

The test results clearly indicated that some proposals exhibited a substantial improvement in compression performance as compared to the corresponding AVC anchors - and, in a number of cases, the performance of the best proposals can be roughly characterized as achieving similar quality when using only half of the bit rate.

All proposals basically used a coding architecture that was conceptually similar to AVC (and prior video coding standards), containing the following basic elements:

• Block-based
• Variable block sizes
• Block motion compensation
• Fractional-pel motion vectors
• Spatial intra prediction
• Spatial transform of residual difference
• Integer-based transform designs
• Arithmetic or VLC-based entropy coding
• In-loop filtering to form final decoded picture

However, there was a large variety of differences at the individual coding tool level.

For further details, please subscribe to the JCT-VC reflector and look for the corresponding email in the archives.