MPEG news: a report from the 103rd meeting, Geneva, Switzerland

MPEG plenary meeting at CICG in Geneva, CH

The 103rd MPEG meeting was held in Geneva, Switzerland, January 21-15, 2013. The official press release can be found here (doc only) and I'd like to introduce the new MPEG-H standard (ISO/IEC 23008) referred to as high efficiency coding and media delivery in heterogeneous environments:

• Part 1: MPEG Media Transport (MMT) - status: 2nd committee draft (CD)
• Part 2: High Efficiency Video Coding (HEVC) - status: final draft international standard (FDIS)
• Part 3: 3D Audio - status: call for proposals (CfP)

MPEG Media Transport (MMT)

The MMT project was started in order to address the needs of modern media transport applications going beyond the capabilities offered by existing means of transportation such as formats defined by MPEG-2 transport stream (M2TS) or ISO base media file format (ISOBMFF) group of standards. The committee draft was approved during the 101st MPEG meeting. As a response to the CD ballot, MPEG received more than 200 comments from national bodies and, thus, decided to issue the 2nd committee draft which will be publicly available by February 7, 2013.

High Efficiency Video Coding (HEVC) - ITU-T H.265 | MPEG HEVC

HEVC is the next generation video coding standard jointly developed by ISO/IEC JTC1/SC29/WG11 (MPEG) and the Video Coding Experts Group (VCEG) of ITU-T WP 3/16. Please note that both ITU-T and ISO/IEC MPEG use the term "high efficiency video coding" in the the title of the standard but one can expect - as with its predecessor - that the former will use ITU-T H.265 and the latter will use MPEG-H HEVC for promoting its standards. If you don't want to participate in this debate, simply use high efficiency video coding.

The MPEG press release says that the "HEVC standard reduces by half the bit rate needed to deliver high-quality video for a broad variety of applications" (note: compared to its predecessor AVC). The editing period for the FDIS goes until March 3, 2013 and then with the final preparations and a 2 month balloting period (yes|no vote only) once can expect the International Standard (IS) to be available early summer 2013. Please note that there are no technical differences between FDIS and IS.

The ITU-T press release describes HEVC as a standard that "will provide a flexible, reliable and robust solution, future-proofed to support the next decade of video. The new standard is designed to take account of advancing screen resolutions and is expected to be phased in as high-end products and services outgrow the limits of current network and display technology."

HEVC currently defines three profiles:

• Main Profile for the "Mass-market consumer video products that historically require only 8 bits of precision".
• Main 10 Profile "will support up to 10 bits of processing precision for applications with higher quality demands".
• Main Still Picture Profile to support still image applications, hence, "HEVC also advances the state-of-the-art for still picture coding"

3D Audio

The 3D audio standard shall complement MMT and HEVC assuming that in a "home theater" system a large number of loudspeakers will be deployed. Therefore, MPEG has issued a Call for Proposals (CfP) with the selection of the reference model v0 due in July 2013. The CfP says that MPEG-H 3D Audio "might be surrounding the user and be situated at high, mid and low vertical positions relative to the user’s ears. The desired sense of audio envelopment includes both immersive 3D audio, in the sense of being able to virtualize sound sources at any position in space, and accurate audio localization, in terms of both direction and distance."

"In addition to a “home theater” audio-visual system, there may be a “personal” system having a tablet-sized visual display with speakers built into the device, e.g. around the perimeter of the display. Alternatively, the personal device may be a hand-held smart phone. Headphones with appropriate spatialization would also be a means to deliver an immersive audio experience for all systems."

Complementary to the CfP, MPEG also provided the encoder input format for MPEG-H 3D audio and a draft MPEG audio core experiment methodology for 3D audio work.

Publicly available MPEG output documents

The following documents shall be come available at http://mpeg.chiariglione.org/ (note: some may have an editing period - YY/MM/DD). If you have difficulties to access one of these documents, please feel free to contact me.

• Study text of DIS of ISO/IEC 23000-13, Augmented Reality Application Format (13/01/25)
• Study text of DTR of ISO/IEC 23000-14, Augmented reality reference model (13/02/25)
• Text of ISO/IEC FDIS 23005-1 2nd edition Architecture (13/01/25)
• Text of ISO/IEC 2nd CD 23008-1 MPEG Media Transport (13/02/07)
• Text of ISO/IEC 23008-2:201x/PDAM1 Range Extensions (13/03/22)
• Text of ISO/IEC 23008-2:201x/PDAM2 Multiview Extensions (13/03/22)
• Call for Proposals on 3D Audio (13/01/25)
• Encoder Input Format for MPEG-H 3D Audio (13/02/08)
• Draft MPEG Audio CE methodology for 3D Audio work (13/01/25)
• Draft Requirements on MPEG User Descriptions (13/02/08)
• Draft Call for Proposals on MPEG User Descriptions (13/01/25)
• Draft Call for Proposals on Green MPEG (13/01/25)
• Context, Objectives, Use Cases and Requirements for Green MPEG (13/01/25)
• White Paper on State of the Art in compression and transmission of 3D Video (13/01/28)
• MPEG Awareness Event Flyer at 104th MPEG meeting in Incheon (13/02/28)

MPEG news: a report from the 99th meeting, San Jose, CA, USA

The official press release is available here and I'd like to highlight two topics from MPEGs' 99th meeting in San Jose, CA, USA:

• HEVC advances to Committee Draft (CD)
• Public workshop on MPEG-H 3D Audio

High-Efficiency Video Coding reaches first formal milestone towards completion

As described in the official press release "ISO/IEC’s Moving Picture Experts Group (MPEG) is pleased to announce the completion of the ISO/IEC committee draft of the High Efficiency Video Coding (HEVC) standard developed by the Joint Collaborative Team on Video Coding (JCT-VC), a joint team between MPEG and the ITU-T’s Video Coding Experts Group (VCEG)". For those who are not familiar with the ISO/IEC standardization process, committee draft (CD) means that the standard is not yet finalized but entering the committee stage which enables national bodies to comment on the standard. That is, changes to HEVC can be only made through national body comments which needs to be registered in due time.

In terms of performance of HEVC one can conclude that the mission is accomplished. Preliminary HM5 vs. AVC subjective performance comparison looks impressive, i.e. > 50% bitrate reduction overall, specifically 67% in HD and 49% for WVGA sequences. Please note that these results are not validated through official verification tests which are usually conducted in a later stage of the standardization process.

From a deployment perspective currently one profile is foreseen which is preliminarily referred to as the "main" profile with a largest coding unit (LCU) between 16x16 and 64x64 and a max. pictures storage capacity always 6 (compared to AVC which is max. 16) among others.

Research issues: in my last report I wrote "the ultimative goal to have a performance gain of more than 50% compared to the predecessor which is AVC". It seems this has been achieved so one might wonder what else needs to be done. In practice, however, there is always space for improvement, right?

The next step in audio coding: MPEG-H 3D Audio

The MPEG-H 3D Audio Workshop attracted more than 100 attendees which followed presentations covering three areas of 3D audio.

1. ATSC 3.0 and the Future of Broadcast Television (FoBTV)
2. 22.2 multichannel sound for Ultra High Definition TV (UHDTV), Next Generation Broadcast Television, and New Heights in Multichannel Sound: Explorations and Considerations
3. Realistic audio representation technologies for UHDTV, backward-compatible 3D audio coding, and innovating beyond 5.1.

The presentations are publicly available here within a single ZIP file. MPEG established an AhG on 3D Audio (and Audio Maintenance) with the following mandates (among others):

• Progress possible use cases, requirements and evaluation methods for 3D Audio 
• Identify test material appropriate for 3D Audio work and a process to make the material available to interested MPEG delegates.

Subscription to the reflector is open to everyone. A possible timeline for part 3 of MPEG-H could mean to have a Call for Proposals (CfP) in July 2012 followed by the evaluation in January 2013, all preliminary, no guarantee.

Finally, the next meeting will be MPEGs' 100th meeting which will include a social event with participation of representatives from ITU, ISO, IEC, and others.

MPEG news: a report from the 98th meeting, Geneva, Switzerland

#MPEG98 report: DASH=IS ✔ CDVS=CfP eval ✔ {MMT, HEVC, 3DAudio}=MPEG-H ✔ IVC={IVC, WebVC} ✔ 3DVC=CfP eval ✔

... MPEG news from its 98th meeting in Geneva, Switzerland with less than 140 characters and a lot of acronyms. The official press release is, as usual, here. As you can see from the press release, MPEG produced significant results, namely:

• MPEG Dynamic Adaptive Streaming over HTTP (DASH) ratified
• 3D Video Coding: Evaluation of responses to Call for Proposals
• MPEG royalty free video coding: Internet Video Coding (IVC) + Web Video Coding (WebVC)
• High Efficiency Coding and Media Delivery in Heterogeneous Environments: MPEG-H comprising MMT, HEVC, 3DAC
• Compact Descriptors for Visual Search (CDVS): Evaluation of responses to the Call for Proposals
• Call for requirements: Multimedia Preservation Description Information (MPDI)
• MPEG Augmented Reality (AR)

As you can see, a long list of achievements within a single meeting but let's dig inside. For each topic I've also tried to provide some research issues which I think are worth to investigate both inside and outside MPEG. 

MPEG Dynamic Adaptive Streaming over HTTP (DASH): DASH=IS ✔

As the official press release states, the MPEG ratifies its draft standard for DASH and it comes better, the standard should become publicly available which I expect to happen somewhat early next year, approx. March 2012, or maybe earlier. I say "should" because there is no guarantee that this will actually happen but signs are good. In the meantime, feel free using our software to play around and we expect to update it to the latest version of the standard as soon as possible. Finally, IEEE Computer Society Computing Now has put together a theme on Video for the Universal Web featuring DASH.

Research issues: performance, bandwidth estimation, request scheduling (aka adaptation logic), and Quality of Service/Experience.

3D Video Coding: 3DVC=CfP eval ✔

MPEG evaluated more than 20 proposals submitted as a response to the call issued back in April 2011. The evaluation of the proposal comprised subjective quality assessments conducted by 13 highly qualified test laboratories distributed around the world and coordinated by the COST Action IC1003 QUALINET. The report of the subjective test results from the call for proposals on 3D video coding will be available by end of this week. MPEG documented the standardization tracks considered in 3DVC (i.e., compatible with MVC, AVC base-view, HEVC, ...) and agreed on a common software based on the best-performing proposals.

Research issues: encoding efficiency of 3D depth maps and compatibility for the various target formats (AVC, MVC, HEVC) as well as depth map estimation at the client side.

MPEG royalty free video coding: IVC vs. WebVC

In addition to the evaluation of the responses to the call for 3DVC, MPEG also evaluated the responses to the Internet Video Coding call. Based on the responses, MPEG decided to follow up with two approaches namely Internet Video Coding (IVC) and Web Video Coding (WebVC). The former - IVC - is based on MPEG-1 technology which is assumed to be royalty-free. However, it requires some performance boosts in order to make it ready for the Internet. MPEG's approach is a common platform called Internet video coding Test Model (ITM) which serves as the basis for further improvements. The latter - WebVC - is based on the AVC constrained baseline profile which performance is well-known and satisfactory but, unfortunately, it is not clear which patents of the AVC patent pool apply to this profile. Hence, a working draft (WD) of WebVC will be provided (also publicly available) in order to get patent statements from companies. The WD will be publicly available by December 19th.

Further information:

• Royalty-Free MPEG Proposals Announced
• MPEG Plus or Patent Pool Lite? MPEG Mulls Royalty-Free Proposals
• Half of MPEG-2 Patents Expire in 2012

Research issues: coding efficiency with using only royalty free coding tools whereby the optimization is first towards royalties and then efficiency.

MPEG-H

A new star is born which is called MPEG-H referred to as "High Efficiency Coding and Media Delivery in Heterogeneous Environments" comprising three parts: Pt. 1 MMT, Pt. 2 HEVC, Pt. 3 3D Audio. There's a document called context and objective of MPEG-H but I can't find out whether it's public (I come back later on this).

Part 1: MMT (MPEG Media Transport) is progressing (slowly) but a next step should be definitely to check the relationship of MMT and DASH for which an Ad-hoc Group has been established (N12395), subscribe here, if interested.
Research issues: very general at the moment, what is the best delivery method (incl. formats) for future multimedia applications? Answer: It depends, ... ;-)

Part 2: HEVC (High-Efficiency Video Coding) made significant progress at the last meeting, in particular: only one entropy coder (note: AVC has two, CABAC and CAVLC which are supported in different profiles), 8 bit decoding (could be also 10 bit, probably done in some profiles), specific integer transform, stabilized and more complete high-level syntax and HRD description (i.e., reference picture buffering, tiles, slices, and wavefronts enabling parallel decoding process). Finally, a prototype has been demonstrated decoding HEVC in software on an iPad 2 at WVGA resolution and the 10min Big Buck Bunny sequence at SD resolution with avg. 800 kbit/s which clearly outperformed the corresponding AVC versions.
Research issues: well, coding efficiency, what else? The ultimative goal to have a performance gain of more than 50% compared to the predecessor which is AVC.

Part 3: 3D Audio Coding (3DAC) is in its early stages but there will be an event during San Jose meeting which will be announced here. As of now, use cases are provided (home theatre, personal TV, smartphone TV, multichannel TV) as well as candidate requirements and evaluation methods. One important aspect seems to be user experience for highly immersive audio (i.e., 22.2, 10.2, 5.1) including bitstream adaptation for low-bandwidth and low-complexity.
Research issues: sorry, I'm not really an audio guy but I assume it's coding efficiency, specifically for 22.2 channels ;-)

Compact Descriptors for Visual Search (CDVS)

For CDVS, responses to the call for proposals (from 10 companies/institutions) have been evaluated and a test model has been established based on the best performing proposals. The next steps include the improvement of the test model towards for inclusion in the MPEG-7 standard.
Research issues: descriptor efficiency for the intended application as well as precision on the information retrieval results.

Multimedia Preservation Description Information (MPDI)

The aim of this new work item is to provide "standard technology helping users to preserve digital multimedia that is used in many different domains, including cultural heritage, scientific research, engineering, education and training, entertainment, and fine arts for long-term across system, organizational, administrative and generational boundaries". It comes along with two public documents, the current requirements and a call for requirements which are due at the 100th MPEG meeting in April 2002.
Research issues: What and how to preserve digital multimedia information?

Augmented Reality (AR)

MPEG's newest project is on Augmented Reality (AR), starting with an application format for which a working draft exists. Furthermore, draft requirements and use cases are available. These three documents will be available on Dec 31st.
Research issues: N/A

Finally, I hope now you can better understand what I've put at the beginning with all these acronyms ...

#MPEG98 report: DASH=IS ✔ CDVS=CfP eval ✔ {MMT, HEVC, 3DAudio}=MPEG-H ✔ IVC={IVC, WebVC} ✔ 3DVC=CfP eval ✔