What happened in multimedia communication in 2018?

In January 2018 I wrote a blog post entitled "What to care about in multimedia communication in 2018?" and I think it's worth looking back to see what actually happened with respect to next generation video coding formats and adaptive streaming techniques.

In April 2018, the responses to the call for proposals for the next standard in video compression have been evaluated and a first working draft and test model for the Versatile Video Coding (VVC) standard have been approved. At this point already, some proposals demonstrated compression efficiency gains of typically 40% or more when compared to using HEVC. Currently, working draft 3 and test model 3 of VVC (VTM 3) are available and we may certainly expect compression efficiency gains well-beyond the targeted 50% for the final standard. An overview about VVC can be found here (by C. Feldmann) and here (by M. Wien). The licensing issues have been acknowledged and, thus, the Media Coding Industry Forum (MC-IF) has been established.

At the beginning of 2018, everyone was also very curious about AOMedia and AV1. Version 1 of the specification has been finally become available and in the meantime it is implemented/deployed on both content provisioning/encoding (e.g. Bitmovin) and content consumption/decoding (e.g., Chrome, Firefox). In this context, we also published a multi-codec DASH dataset comprising AVC, HEVC, VP9, and AV1 (VVC will be added at a later stage). In general, however, we are entering the era of multiple video codecs deployed in products and services whereby this trend is also confirmed by Bitmovin's latest video developer survey.

MPEG-DASH 3rd edition has been approved and is awaiting publication but I expect this to happen in 2019 though. An overview of the MPEG-DASH status is shown in the figure below.

In this context, the DASH-IF produced various vital assets such as interoperability guidelines (latest v4.3, content protection, ATSC 3.0, SAND), test vectors, conformance tools, and a reference client. For informative aspects of MPEG-DASH such as the bitrate adaptation schemes the interested reader is referred to our survey. This survey gives an overview about existing techniques (see figure below) and also outlines future research. It is available for free for everyone (open access).

Finally, I mentioned a couple of scientific events in 2018 including QoMEX, MMSys (NOSSDAV, PV), ICME, ICIP, PCS, and MIPR. I have attended all of the them (except PCS), each showing advances in their respective field. These events are probably worth to attend also in 2019 but I will certainly blog about this early next year. However, I'd like to hear your opinion of what happened in 2018 and what we may expect in 2019...

Packet Video 2018: Best Paper Award

Packet Video 2018 -- Best Paper Award

https://2018.packet.video/

The TPC chairs of Packet Video 2018 identified the following candidates based on the reviews:

• A. Zare, A. Aminlou, M. Hannuksela, 6K Effective Resolution with 4K HEVC Decoding Capability for OMAF-compliant 360° Video Streaming
• J. Schneider, M. Bläser, M. Wien, Sparse Coding based Frequency Adaptive Loop Filtering for Video Coding
• S. Arisu, A. Begen, Quickly Starting Media Streams Using QUIC

... and the winner of the Packet Video 2018 Best Paper Award is:

6K Effective Resolution with 4K HEVC Decoding Capability for OMAF-compliant 360° Video Streaming

A. Zare, A. Aminlou, M. Hannuksela

Abstract: The recent Omnidirectional MediA Format (OMAF) standard specifies delivery of 360° video content. OMAF supports only equirectangular (ERP) and cubemap projections and their region- wise packing with a limitation on video decoding capability to the maximum resolution of 4K (e.g., 4096x2048). Streaming of 4K ERP content allows only a limited viewport resolution, which is lower than the resolution of many current head-mounted displays (HMDs). In order to take the full advantage of those HMDs, this work proposes a specific mixed-resolution packing of 6K (6144x3072) ERP content and its realization in tile-based streaming, while complying with the 4K-decoding constraint and the High Efficiency Video Coding (HEVC) standard. Experimental results indicate that, using Zonal-PSNR test methodology, the proposed layout decreases the streaming bitrate up to 32% in terms of BD-rate, when compared to mixed-quality viewport-adaptive streaming of 4K ERP as an alternative solution.

T. Schierl (left), A. Zare (middle), R. Zimmermann (right)

T. Schierl (left), A. Zare (middle), R. Zimmermann (right)

Packet Video 2018: Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Streaming

Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Streaming

Armin Trattnig (Bitmovin Inc.), Christian Timmerer (Alpen-Adria-Universität Klagenfurt / Bitmovin Inc.), and Christopher Mueller (Bitmovin Inc.)

[PDF]

Abstract: About 300 hours of video are uploaded to YouTube every minute. The main technology to delivery YouTube content to various clients is HTTP adaptive streaming and the majority of today’s internet traffic comprises streaming audio and video. In this paper, we investigate content provisioning for HTTP adaptive streaming under predefined aspects representing content features and upload characteristics as well and apply it to YouTube. Additionally, we compare the YouTube’s content upload and processing functions with a commercially available video encoding service. The results reveal insights into YouTube’s content upload and processing functions and the methodology can be applied to similar services. All experiments conducted within the paper allow for reproducibility thanks to the usage of open source tools, publicly available datasets, and scripts used to conduct the experiments on virtual machines.

Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Streaming from Christian Timmerer

QoMEX, MMSys, PCS, ICME, ICIP (Re: What to care about in multimedia communication in 2018?)

In January 2018 I posted about "What to care about in multimedia communication in 2018?" and I think it's time to see what happened so far.

MPEG’ & VCEG’s new video coding project starts strong and is now called Versatile Video Coding (VVC) -- see here a presentation about VVC at Alpen-Adria-Universität Klagenfurt. At the same time AOMedia "releases AV1" with many showcases at NAB2018.

In my previous blog post I stated that PCS and ICIP will be full of papers related to VVC and AV1. The program of PCS is available now, which even has session on "Performance Comparison of Video Standards" and many other interesting things. ICIP program is not yet available but I expect similar papers (although MPEG/VCEG meeting is scheduled in parallel to ICIP).

Also, our multi-codec DASH dataset paper at MMSys addresses AV1 and check out the program, including also 23rd Packet Video Workshop, which is about new media applications, content coding, and media streaming. QoMEX is actually very close and its program is available here (incl. a subjective comparison of HEVC/H.265, VP9, and AV1).

Once all these papers are available, it would be interesting to perform an in-depth analysis and comparison thereof.

ICME program is not yet available but has a tutorial on "Trends and Recent Developments in Video Coding Standardization" (btw. you may also come and see our tutorial on "Delivering Traditional and Omnidirectional Media").

If you want to get into this area, here are the most important dates:

• QoMEX, Sardinia, Italy, May 29 - June 1, 2018
• MMSys (incl. PV), Amsterdam, The Netherlands, June 12-15, 2018
• PCS, San Francisco, CA, USA, June 24-27, 2018
• MPEG-Meeting, Ljubljana, Slovenia, Jule 16-20, 2018
• ICME, San Diego, CA, USA, July 23-27, 2018
• ICIP, Athens, Greece, October 7-10, 2018

Finally, MPEG celebrated 30 years on May 10, 2018 and you may want to read Leonardo's blog post or the last paragraph of my MPEG report.

What to care about in multimedia communication in 2018?

In the past days/weeks you may have witness a high number of forecasts/predictions for 2018, like this one here. I'm not so good at predictions and we all learned these days to be careful about speculation. Thus, my focus here is on things to care about in 2018.

MPEG and VCEG are working towards a new video coding standard (naming, number scheme yet to be defined) and the call for proposals is out. Responses will be evaluated by the 122nd MPEG meeting in April 2018 (San Diego, CA, USA) and a new standard is expected to be available in late 2020. The main focus of the CfP is (i) 360-degree omnidirectional video, (ii) high-dynamic range (HDR), (iii) wide colour gamut (WCG), and (iv) conventional standard-dynamic-range camera content. The goal is -- simple, as usual -- compress digital video content, i.e., twice as much as you did before with the same video quality, e.g., as HEVC, or get higher quality with the same number of bits (or a combination thereof). Initial, preliminary results indicate this goal is feasible and everyone is looking forward to the MPEG meeting in April; certainly a place to be.

In addition to what MPEG/VCEG is doing, the Alliance for Open Media gained significant attention with its AV1 codec, first demos are available, and recently also Apple joined AOM. AV1 is not longer controlled by a single company, and, thus, it is becoming a real alternative in the video coding landscape, specifically for the streaming market. The nice thing, it's open source and royalty-free! In other words, one should not neglect AV1 and I think we will see many, hopefully good news in 2018.

On this topic, you might be interested in reading this and this.

What about DASH in 2018? We will see a 3rd edition of MPEG-DASH, the DASH-IF will further work on interoperability points, and I expect further convergence of DASH and HLS towards CMAF. However, I also expect minor changes for the main, common use cases utilizing the core technology of HTTP adaptive streaming. Changes, if any, will be transparent to most of us. On the other hand, immersive media and user engagement will become more and more important as more services are delivered over the top leading to more content becoming available to end users, thus, increasing competition among providers, vendors, etc. As a consequence, (a) content, (b) quality, and (c) costs will be important aspects, whereby (a+c) are "easy to sell" but (b) is still difficult to quantify (and "sell") with many open issues to solve in the (near) future.

Before drifting off to forecasts and predictions, I'd like to conclude with a list of scientific events in 2018 which are worthwhile to attend:

• QoMEX -- Int'l Conf. on Quality of Multimedia Experience -- will be hosted in Sardinia, Italy from May 29-31, which is THE conference to be for QoE of multimedia applications and services. Submission deadline is January 15/22, 2018.
• MMSys -- Multimedia Systems Conf. -- and specifically Packet Video, which will be on June 12 in Amsterdam, The Netherlands. Packet Video is THE adaptive streaming scientific event 2018. Submission deadline is March 1, 2018.
• Additionally, you might be interested in ICME (July 23-27, 2018, San Diego, USA; I'm part of a tutorial there;), ICIP (October 7-10, 2018, Athens, Greece; specifically in the context of video coding), and PCS (June 24-27, 2018, San Francisco, CA, USA; also in the context of video coding).
• The DASH-IF academic track hosts special events at MMSys (Excellence in DASH Award) and ICME (DASH Grand Challenge).
• MIPR -- 1st Int'l Conf. on Multimedia Information Processing and Retrieval -- will be in Miami, Florida, USA from April 10-12, 2018. It has a broad range of topics including networking for multimedia systems as well as systems and infrastructures.

MPEG news: a report from the 120th meeting, Macau, China

MPEG Meeting Plenary

The original blog post can be found at the Bitmovin Techblog and has been updated here to focus on and highlight research aspects. Additionally, this version of the blog post will be also posted at ACM SIGMM Records.

MPEG News Archive

The MPEG press release comprises the following topics:

• Point Cloud Compression – MPEG evaluates responses to call for proposal and kicks off its technical work 
• The omnidirectional media format (OMAF) has reached its final milestone 
• MPEG-G standards reach Committee Draft for compression and transport technologies of genomic data 
• Beyond HEVC – The MPEG & VCEG call to set the next standard in video compression 
• MPEG adds better support for mobile environment to MMT 
• New standard completed for Internet Video Coding 
• Evidence of new video transcoding technology using side streams 

Point Cloud Compression

At its 120th meeting, MPEG analysed the technologies submitted by nine industry leaders as responses to the Call for Proposals (CfP) for Point Cloud Compression (PCC). These technologies address the lossless or lossy coding of 3D point clouds with associated attributes such as colour and material properties. Point clouds are referred to as unordered sets of points in a 3D space and typically captured using various setups of multiple cameras, depth sensors, LiDAR scanners, etc., but can also be generated synthetically and are in use in several industries. They have recently emerged as representations of the real world enabling immersive forms of interaction, navigation, and communication. Point clouds are typically represented by extremely large amounts of data providing a significant barrier for mass market applications. Thus, MPEG has issued a Call for Proposal seeking technologies that allow reduction of point cloud data for its intended applications. After a formal objective and subjective evaluation campaign, MPEG selected three technologies as starting points for the test models for static, animated, and dynamically acquired point clouds. A key conclusion of the evaluation was that state-of-the-art point cloud compression can be significantly improved by leveraging decades of 2D video coding tools and combining 2D and 3D compression technologies. Such an approach provides synergies with existing hardware and software infrastructures for rapid deployment of new immersive experiences.

Although the initial selection of technologies for point cloud compression has been concluded at the 120th MPEG meeting, it could be also seen as a kick-off for its scientific evaluation and various further developments including the optimization thereof. It is expected that various scientific conference will focus on point cloud compression and may open calls for grand challenges like for example at IEEE ICME 2018.

Omnidirectional Media Format (OMAF)

The understanding of the virtual reality (VR) potential is growing but market fragmentation caused by the lack of interoperable formats for the storage and delivery of such content stifles VR’s market potential. MPEG’s recently started project referred to as Omnidirectional Media Format (OMAF) has reached Final Draft of International Standard (FDIS) at its 120th meeting. It includes

• equirectangular projection and cubemap projection as projection formats; 
• signalling of metadata required for interoperable rendering of 360-degree monoscopic and stereoscopic audio-visual data; and 
• provides a selection of audio-visual codecs for this application. 

It also includes technologies to arrange video pixel data in numerous ways to improve compression efficiency and reduce the size of video, a major bottleneck for VR applications and services, The standard also includes technologies for the delivery of OMAF content with MPEG-DASH and MMT.

MPEG has defined a format comprising a minimal set of tools to enable interoperability among implementers of the standard. Various aspects are deliberately excluded from the normative parts to foster innovation leading to novel products and services. This enables us -- researcher and practitioners -- to experiment with these new formats in various ways and focus on informative aspects where typically competition can be found. For example, efficient means for encoding and packaging of omnidirectional/360-degree media content and its adaptive streaming including support for (ultra-)low latency will become a big issue in the near future.

MPEG-G: Compression and Transport Technologies of Genomic Data

The availability of high throughput DNA sequencing technologies opens new perspectives in the treatment of several diseases making possible the introduction of new global approaches in public health known as “precision medicine”. While routine DNA sequencing in the doctor’s office is still not current practice, medical centers have begun to use sequencing to identify cancer and other diseases and to find effective treatments. As DNA sequencing technologies produce extremely large amounts of data and related information, the ICT costs of storage, transmission, and processing are also very high. The MPEG-G standard addresses and solves the problem of efficient and economical handling of genomic data by providing new

• compression technologies (ISO/IEC 23092-2) and 
• transport technologies (ISO/IEC 23092-1), 

which reached Committee Draft level at its 120th meeting.

Additionally, the Committee Drafts for

• metadata and APIs (ISO/IEC 23092-3) and 
• reference software (ISO/IEC 23092-4) 

are scheduled for the next MPEG meeting and the goal is to publish Draft International Standards (DIS) at the end of 2018.

This new type of (media) content, which requires compression and transport technologies, is emerging within the multimedia community at large and, thus, input is welcome.

Beyond HEVC – The MPEG & VCEG Call to set the Next Standard in Video Compression

The 120th MPEG meeting marked the first major step toward the next generation of video coding standard in the form of a joint Call for Proposals (CfP) with ITU-T SG16’s VCEG. After two years of collaborative informal exploration studies and a gathering of evidence that successfully concluded at the 118th MPEG meeting, MPEG and ITU-T SG16 agreed to issue the CfP for future video coding technology with compression capabilities that significantly exceed those of the HEVC standard and its current extensions. They also formalized an agreement on formation of a joint collaborative team called the “Joint Video Experts Team” (JVET) to work on development of the new planned standard, pending the outcome of the CfP that will be evaluated at the 122nd MPEG meeting in April 2018. To evaluate the proposed compression technologies, formal subjective tests will be performed using video material submitted by proponents in February 2018. The CfP includes the testing of technology for 360° omnidirectional video coding and the coding of content with high-dynamic range and wide colour gamut in addition to conventional standard-dynamic-range camera content. Anticipating a strong response to the call, a “test model” draft design is expected be selected in 2018, with development of a potential new standard in late 2020.

The major goal of a new video coding standard is to be better than its successor (HEVC). Typically this "better" is quantified by 50% which means, that it should be possible encode the video at the same quality with half of the bitrate or a significantly higher quality with the same bitrate including. However, at this time the “Joint Video Experts Team” (JVET) from MPEG and ITU-T SG16 faces competition from the Alliance for Open Media, which is working on AV1. In any case, we are looking forward to an exciting time frame from now until this new codec is ratified and how it will perform compared to AV1. Multimedia systems and applications will also benefit from new codecs which will gain traction as soon as first implementations of this new codec becomes available (note that AV1 is available as open source already and continuously further developed).

MPEG adds Better Support for Mobile Environment to MPEG Media Transport (MMT)

MPEG has approved the Final Draft Amendment (FDAM) to MPEG Media Transport (MMT; ISO/IEC 23008-1:2017), which is referred to as “MMT enhancements for mobile environments”. In order to reflect industry needs on MMT, which has been well adopted by broadcast standards such as ATSC 3.0 and Super Hi-Vision, it addresses several important issues on the efficient use of MMT in mobile environments. For example, it adds distributed resource identification message to facilitate multipath delivery and transition request message to change the delivery path of an active session. This amendment also introduces the concept of a MMT-aware network entity (MANE), which might be placed between the original server and the client, and provides a detailed description about how to use it for both improving efficiency and reducing delay of delivery. Additionally, this amendment provides a method to use WebSockets to setup and control an MMT session/presentation.

New Standard Completed for Internet Video Coding

A new standard for video coding suitable for the internet as well as other video applications, was completed at the 120th MPEG meeting. The Internet Video Coding (IVC) standard was developed with the intention of providing the industry with an “Option 1” video coding standard. In ISO/IEC language, this refers to a standard for which patent holders have declared a willingness to grant licenses free of charge to an unrestricted number of applicants for all necessary patents on a worldwide, non-discriminatory basis and under other reasonable terms and conditions, to enable others to make, use, and sell implementations of the standard. At the time of completion of the IVC standard, the specification contained no identified necessary patent rights except those available under Option 1 licensing terms. During the development of IVC, MPEG removed from the draft standard any necessary patent rights that it was informed were not available under such Option 1 terms, and MPEG is optimistic of the outlook for the new standard. MPEG encourages interested parties to provide information about any other similar cases. The IVC standard has roughly similar compression capability as the earlier AVC standard, which has become the most widely deployed video coding technology in the world. Tests have been conducted to verify IVC’s strong technical capability, and the new standard has also been shown to have relatively modest implementation complexity requirements.

Evidence of new Video Transcoding Technology using Side Streams

Following a “Call for Evidence” (CfE) issued by MPEG in July 2017, evidence was evaluated at the 120th MPEG meeting to investigate whether video transcoding technology has been developed for transcoding assisted by side data streams that is capable of significantly reducing the computational complexity without reducing compression efficiency. The evaluations of the four responses received included comparisons of the technology against adaptive bit-rate streaming using simulcast as well as against traditional transcoding using full video re-encoding. The responses span the compression efficiency space between simulcast and full transcoding, with trade-offs between the bit rate required for distribution within the network and the bit rate required for delivery to the user. All four responses provided a substantial computational complexity reduction compared to transcoding using full re-encoding. MPEG plans to further investigate transcoding technology and is soliciting expressions of interest from industry on the need for standardization of such assisted transcoding using side data streams.

MPEG currently works on two related topics which are referred to as network-distributed video coding (NDVC) and network-based media processing (NBMP). Both activities involve the network, which is more and more evolving to highly distributed compute and delivery platform as opposed to a bit pipe, which is supposed to deliver data as fast as possible from A to B. This phenomena could be also interesting when looking at developments around 5G, which is actually much more than just radio access technology. These activities are certainly worth to monitor as it basically contributes in order to make networked media resources accessible or even programmable. In this context, I would like to refer the interested reader to the December'17 theme of the IEEE Computer Society Computing Now, which is about Advancing Multimedia Content Distribution.

Publicly available documents from the 120th MPEG meeting can be found here (scroll down to the end of the page). The next MPEG meeting will be held in Gwangju, Korea, January 22-26, 2018. Feel free to contact Christian Timmerer for any questions or comments.

Some of the activities reported above are considered within the Call for Papers at 23rd Packet Video Workshop (PV 2018) co-located with ACM MMSys 2018 in Amsterdam, The Netherlands. Topics of interest include (but are not limited to):

• Adaptive media streaming, and content storage, distribution and delivery 
• Network-distributed video coding and network-based media processing 
• Next-generation/future video coding, point cloud compression 
• Audiovisual communication, surveillance and healthcare systems 
• Wireless, mobile, IoT, and embedded systems for multimedia applications 
• Future media internetworking: information-centric networking and 5G 
• Immersive media: virtual reality (VR), augmented reality (AR), 360° video and multi-sensory systems, and its streaming 
• Machine learning in media coding and streaming systems 
• Standardization: DASH, MMT, CMAF, OMAF, MiAF, WebRTC, MSE, EME, WebVR, Hybrid Media, WAVE, etc.
• Applications: social media, game streaming, personal broadcast, healthcare, industry 4.0, education, transportation, etc. 

Important dates

• Submission deadline: March 1, 2018 
• Acceptance notification: April 9, 2018 
• Camera-ready deadline: April 19, 2018

Packet Video Workshop 2018

23rd Packet Video Workshop 2018

June 12, 2018, Amsterdam, The Netherlands

(co-located with ACM MMSys'18)

https://2018.packet.video/

Workshop Co-Chairs

• Ali C. Begen, Ozyegin University / Networked Media, Turkey (ali.begen at networked.media)
• Christian Timmerer, Alpen-Adria-Universität Klagenfurt / Bitmovin Inc., Austria (christian.timmerer at itec.uni-klu.ac.at)

Workshop TPC Co-Chairs

• Roger Zimmermann, National University of Singapore (NUS), Singapore (rogerz at comp.nus.edu.sg)
• Thomas Schierl, Fraunhofer Heinrich Hertz Institute (HHI), Germany (thomas.schierl at hhi.fraunhofer.de)

The 23rd Packet Video Workshop (PV 2018) is devoted to presenting technological advancements and innovations in video and multimedia transmission over packet networks. The workshop provides a unique venue for people from the media coding and networking fields to meet, interact and exchange ideas. Its charter is to promote the research and development in both established and emerging areas of video streaming and multimedia networking. PV 2018 will be held in Amsterdam on June 12th. The workshop will be a single-track event and welcomes paper submissions from both cutting-edge research, and business and consumer applications. PV 2018 will be co-located with ACM MMSys, NOSSDAV, NetGames and MMVE.

PV 2018 seeks papers in all areas of media delivery over current IP and future networks. Authors are especially encouraged to submit papers with real-world experimental results and datasets.

Topics of interest include (but are not limited to)

• Adaptive media streaming, and content storage, distribution and delivery
• Network-distributed video coding and network-based media processing
• Next-generation/future video coding, point cloud compression
• Audiovisual communication, surveillance and healthcare systems
• Wireless, mobile, IoT, and embedded systems for multimedia applications
• Future media internetworking: information-centric networking and 5G
• Immersive media: virtual reality (VR), augmented reality (AR), 360° video and multi-sensory systems, and its streaming
• Machine learning in media coding and streaming systems
• Standardization: DASH, MMT, CMAF, OMAF, MiAF, WebRTC, MSE, EME, WebVR, Hybrid Media, WAVE, etc.
• Applications: social media, game streaming, personal broadcast, healthcare, industry 4.0, education, transportation, etc.

Important dates

• Submission deadline: March 1, 2018
• Acceptance notification: April 9, 2018
• Camera-ready deadline: April 19, 2018

Submission instructions
Prospective authors are invited to submit an electronic version of full papers, in PDF format, up to six printed pages in length (double column ACM conference format) at the PV 2018 Web site. The authors are also encouraged to regularly check the PV 2018 web site for the latest information and updates. The proceedings will be published by ACM Digital Library.