DCC’20: Fast Multi-Rate Encoding for Adaptive HTTP Streaming

Fast Multi-Rate Encoding for Adaptive HTTP Streaming

Data Compression Conference 2020, March 24 – 27, Cliff Lodge, Snowbird, UT

[PDF] (coming soon)

Hadi Amirpour, Ekrem Çetinkaya (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), and Mohammad Ghanbari (University of Tehran, University of Essex)

Abstract: Adaptive HTTP streaming is the preferred method to deliver multimedia content on the internet. It provides multiple representations of the same content in different qualities (i.e., bit-rates and resolutions) and allows the client to request segments from the available representations in a dynamic, adaptive way depending on its context. The growing number of representations in adaptive HTTP streaming makes encoding of one video segment at different representations a challenging task in terms of encoding time-complexity. In this paper, information of both the highest and lowest quality representations are used to limit Rate-Distortion Optimization (RDO) for each Coding Unit Tree (CTU) in High Efficiency Video Coding. Our proposed method first encodes the highest quality representation and consequently uses it to encode the lowest quality representation. In particular, the block structure and the selected reference frame of both the highest and lowest quality representations are then used to predict and shorten the RDO process of each CTU for intermediate quality representations. Our proposed method introduces a delay of two CTUs thanks to employing parallel processing techniques. Experimental results show a significant reduction in time-complexity over the reference software (38%) and state-of-the-art (10%) is achieved while quality degradation is negligible.

Keywords:  HTTP adaptive streaming, Multi-rate encoding, HEVC, Fast block partitioning

Video Developer Report 2019

... and Bitmovin did it again; published the 2019 Video Developer Report last week. I've briefly reported about it last year here. Interestingly, this year 542 people from 108 countries participated (vs. 456 from over 67 countries last year).

The biggest challenges seem to be latency (54%) and playback on all devices (41%). Other challenges (>20%) are related to DRM, CDN, user engagement with video, and ads in general.

Last year I've also shared the codec usage and it's probably interesting to compare these numbers with this year's results as shown below. Interestingly, the numbers (for 'planning to implement') are a bit lower compared to last year which could be explained by a more conservative approach from developers or simply by the fact that more people responded to the survey with a greater diversity in terms of different countries.

Current Video Codec Usage and Plans to Implement in next 12 Months.

The actual video codec usage compares to last year's report as follows: AVC (-1), HEVC (+1), VP9 (+/- 0), AV1 (+1).

Another interesting aspect is the usage of streaming formats and plans to implement them within the next 12 months as shown below. Comparing with last year's report (available here), we can observe the following major changes: HLS (-3), MPEG-DASH (-3), RTMP (-2), Smooth Streaming (+2), Progressive Streaming (-1), MPEG-CMAF (+2), HDS (-4).

Current Streaming Formats and Plans to Implement in next 12 Months.

In general, one can observe that the adoption of new formats are happening at a slower pace than expected and I am wondering what this means for the new video coding formats coming up like VVC et al. (note: these are results from a public survey with different participants compared to last years which need to be taken into account when comparing results over years).

For more details, the full report can be downloaded for free from here.

Post-Doctoral Research Positions in "Adaptive Streaming over HTTP and Emerging Networked Multimedia Services"

July 20, 2020: closed; I'm no longer accepting applications.

The Institute of Information Technology at the Alpen-Adria-Universität Klagenfurt invites applications for:

Post-Doctoral Research Positions (100% employment) 

within the Christian Doppler (CD) Pilot Laboratory ATHENA 

“Adaptive Streaming over HTTP and Emerging Networked Multimedia Services” 

at the Faculty for Technical Sciences. The monthly salary for these positions is according to the standard salaries of the Austrian collective agreement, min. € 3.889,50 pre-tax (14x yearly) (Uni-KV: B1 lit.b, http://www.aau.at/en/uni-kv).

ATHENA stands for Adaptive Streaming over HTTP and Emerging Networked Multimedia Services and has been jointly proposed as a CD Laboratory (https://www.cdg.ac.at/) by the Institute of Information Technology (ITEC; http://itec.aau.at) at Alpen-Adria-Universität Klagenfurt (AAU) and Bitmovin GmbH (https://bitmovin.com) to address current and future research and deployment challenges of HTTP Adaptive Streaming (HAS) and emerging streaming methods.

AAU (ITEC) has been working on adaptive video streaming for more than a decade, has a proven record of successful research projects and publications in the field, and has been actively contributing to MPEG standardization for many years, including MPEG-DASH; Bitmovin is a video streaming software company founded by ITEC researchers in 2013 and has developed highly successful, global R&D and sales activities and a world-wide customer base since then.

The aim of ATHENA CD Lab is to research and develop novel paradigms, approaches, (prototype) tools and evaluation results for the areas (1) multimedia content provisioning (i.e., video coding), (2) content delivery (i.e., multimedia networking) and (3) content consumption (i.e., HAS player aspects) in the media delivery chain as well as for (4) end-to-end aspects, with a focus on, but not being limited to, HTTP Adaptive Streaming (HAS). The new approaches and insights are to enable Bitmovin to build innovative applications and services to account for the steadily increasing and changing multimedia traffic on the Internet. In addition, according to the CD Lab model of “application-oriented basic research”, the goal is to publish the results in international, high quality professional journals and conference proceedings.

Your profile:

• Doctoral degree of Technical Science in the field of Computer Science or Electrical Engineering, completed at a domestic or foreign university (with good final degrees);
• Interest and experience in at least on of the following areas: (1) multimedia content provisioning (i.e., video coding) and (2) end-to-end aspects, with a focus on, but not being limited to, HTTP Adaptive Streaming (HAS);
• Excellent English skills, both in written and oral form. 

Desirable qualifications include:

• Excellent programming skills, especially C, C++, Java, JavaScript;
• Relevant international and practical work experience;
• Social and communicative competences and ability to work in a team;
• Experience with university teaching and research activities.

The working language and the research program are in English. There is no need to learn German for this position unless the applicant wants to participate in undergraduate teaching which is optional.

Submit all relevant documents, including copies of all school certificates and performance records, by email to:

Dr. Christian Timmerer
Institute of Information Technology, Alpen-Adria-Universität Klagenfurt
Universitätsstraße 65 – 67, 9020 Klagenfurt, Austria
Email: christian(dot)timmerer(at)itec(dot)aau(dot)at
URL: http://blog.timmerer.com, http://itec.aau.at/, https://athena.itec.aau.at/

Klagenfurt, situated at the beautiful Lake Wörthersee – one of the largest and warmest alpine lakes in Europe – has nearly 100.000 inhabitants. Being a small city, with a Renaissance-style city center reflecting 800 years of history and with Italian influence, Klagenfurt is a pleasant place to live and work. The university is located only about 1.5 kilometers east of Lake Wörthersee and about 3 kilometers west of the city enter.

Doctoral Student Positions in "Adaptive Streaming over HTTP and Emerging Networked Multimedia Services"

The Institute of Information Technology at the Alpen-Adria-Universität Klagenfurt invites applications for:

Dec 29, 2019: closed; I'm no longer searching for doctoral students...

Doctoral Student Positions (100% employment) 

within the Christian Doppler (CD) Pilot Laboratory ATHENA 

“Adaptive Streaming over HTTP and Emerging Networked Multimedia Services” 

at the Faculty for Technical Sciences. The monthly salary for these positions is according to the standard salaries of the Austrian collective agreement, min. € 2,864.50 pre-tax (14x yearly) (Uni-KV: B1, http://www.aau.at/en/uni-kv). Expected start date of employment is September 1st 2019.

ATHENA stands for Adaptive Streaming over HTTP and Emerging Networked Multimedia Services and has been jointly proposed as a CD Laboratory (https://www.cdg.ac.at/) by the Institute of Information Technology (ITEC; http://itec.aau.at) at Alpen-Adria-Universität Klagenfurt (AAU) and Bitmovin GmbH (https://bitmovin.com) to address current and future research and deployment challenges of HTTP Adaptive Streaming (HAS) and emerging streaming methods.

AAU (ITEC) has been working on adaptive video streaming for more than a decade, has a proven record of successful research projects and publications in the field, and has been actively contributing to MPEG standardization for many years, including MPEG-DASH; Bitmovin is a video streaming software company founded by ITEC researchers in 2013 and has developed highly successful, global R&D and sales activities and a world-wide customer base since then.

The aim of ATHENA CD Lab is to research and develop novel paradigms, approaches, (prototype) tools and evaluation results for the areas (1) multimedia content provisioning (i.e., video coding), (2) content delivery (i.e., multimedia networking) and (3) content consumption (i.e., HAS player aspects) in the media delivery chain as well as for (4) end-to-end aspects, with a focus on, but not being limited to, HTTP Adaptive Streaming (HAS). The new approaches and insights are to enable Bitmovin to build innovative applications and services to account for the steadily increasing and changing multimedia traffic on the Internet. In addition, according to the CD Lab model of “application-oriented basic research”, the goal is to publish the results in international, high quality professional journals and conference proceedings.

Your profile:

• Master or diploma degree of Technical Science in the field of Computer Science or Electrical Engineering, completed at a domestic or foreign university (with good final degrees);
• Interest and experience in one or more of the above identified areas, namely (1) multimedia content provisioning (i.e., video coding), (2) content delivery (i.e., multimedia networking) and (3) content consumption (i.e., HAS player aspects) in the media delivery chain as well as (4) end-to-end aspects, with a focus on, but not being limited to, HTTP Adaptive Streaming (HAS);
• Excellent English skills, both in written and oral form. 

Desirable qualifications include:

• Excellent programming skills, especially C, C++, Java, JavaScript;
• Relevant international and practical work experience;
• Social and communicative competences and ability to work in a team;
• Experience with university teaching and research activities.

The working language and the research program are in English. There is no need to learn German for this position unless the applicant wants to participate in undergraduate teaching which is optional.

Submit all relevant documents, including copies of all school certificates and performance records, by email to:

Dr. Christian Timmerer
Institute of Information Technology, Alpen-Adria-Universität Klagenfurt
Universitätsstraße 65 – 67, 9020 Klagenfurt, Austria
Email: christian(dot)timmerer(at)itec(dot)aau(dot)at
URL: http://blog.timmerer.com, http://itec.aau.at/, https://athena.itec.aau.at/

Klagenfurt, situated at the beautiful Lake Wörthersee – one of the largest and warmest alpine lakes in Europe – has nearly 100.000 inhabitants. Being a small city, with a Renaissance-style city center reflecting 800 years of history and with Italian influence, Klagenfurt is a pleasant place to live and work. The university is located only about 1.5 kilometers east of Lake Wörthersee and about 3 kilometers west of the city enter.

MPEG news: a report from the 108th meeting, Valencia, Spain

This blog post is also available at bitmovin tech blog and SIGMM records.

The 108th MPEG meeting was held at the Palacio de Congresos de Valencia in Spain featuring the following highlights (no worries about the acronyms, this is on purpose and they will be further explained below):

• Requirements: PSAF, SCC, CDVA
• Systems: M2TS, MPAF, Green Metadata
• Video: CDVS, WVC, VCB
• JCT-VC: SHVC, SCC
• JCT-3D: MV/3D-HEVC, 3D-AVC
• Audio: 3D audio 

Opening Plenary of the 108th MPEG meeting in Valencia, Spain.

The official MPEG press release can be downloaded from the MPEG Web site. Some of the above highlighted topics will be detailed in the following and, of course, there’s an update on DASH-related matters at the end.

As indicated above, MPEG is full of (new) acronyms and in order to become familiar with those, I’ve put them deliberately in the overview but I will explain them further below.

PSAF – Publish/Subscribe Application Format

Publish/subscribe corresponds to a new network paradigm related to content-centric networking (or information-centric networking) where the content is addressed by its name rather than location. An application format within MPEG typically defines a combination of existing MPEG tools jointly addressing the needs for a given application domain, in this case, the publish/subscribe paradigm. The current requirements and a preliminary working draft are publicly available.

SCC – Screen Content Coding

I’ve introduced this topic in my previous report and this meeting the responses to the CfP have been evaluated. In total, seven responses have been received which meet all requirements and, thus, the actual standardization work is transferred to JCT-VC. Interestingly, the results of the CfP are publicly available. Within JCT-VC, a first test model has been defined and core experiments have been established. I will report more on this as an output of the next meetings…

CDVA – Compact Descriptors for Video Analysis

This project has been renamed from compact descriptors for video search to compact descriptors for video analysis and comprises a publicly available vision statement. That is, interested parties are welcome to join this new activity within MPEG.

M2TS – MPEG-2 Transport Stream

At this meeting, various extensions to M2TS have been defined such as transport of multi-view video coding depth information and extensions to HEVC, delivery of timeline for external data as well as carriage of layered HEVC, green metadata, and 3D audio. Hence, M2TS is still very active and multiple amendments are developed in parallel.

MPAF – Multimedia Preservation Application Format

The committee draft for MPAF has been approved and, in this context, MPEG-7 is extended with additional description schemes.

Green Metadata

Well, this standard does not have its own acronym; it’s simply referred to as MPEG-GREEN. The draft international standard has been approved and national bodies will vote on it at the JTC 1 level. It basically defines metadata to allow clients operating in an energy-efficient way. It comes along with amendments to M2TS and ISOBMFF that enable the carriage and storage of this metadata.

CDVS – Compact Descriptors for Visual Search

CDVS is at DIS stage and provide improvements on global descriptors as well as non-normative improvements of key-point detection and matching in terms of speedup and memory consumption. As all standards at DIS stage, national bodies will vote on it at the JTC 1 level. 

What’s new in the video/audio-coding domain?

• WVC – Web Video Coding: This project reached final draft international standard with the goal to provide a video-coding standard for Web applications. It basically defines a profile of the MPEG-AVC standard including those tools not encumbered by patents.
• VCB – Video Coding for Browsers: The committee draft for part 31 of MPEG-4 defines video coding for browsers and basically defines VP8 as an international standard. This is explains also the difference to WVC.
• SHVC – Scalable HEVC extensions: As for SVC, SHVC will be defined as an amendment to HEVC providing the same functionality as SVC, scalable video coding functionality.
• MV/3D-HEVC, 3D-AVC: These are multi-view and 3D extensions for the HEVC and AVC standards respectively.
• 3D Audio: Also, no acronym for this standard although I would prefer 3DA. However, CD has been approved at this meeting and the plan is to have DIS at the next meeting. At the same time, the carriage and storage of 3DA is being defined in M2TS and ISOBMFF respectively. 

Finally, what’s new in the media transport area, specifically DASH and MMT?

As interested readers know from my previous reports, DASH 2nd edition has been approved has been approved some time ago. In the meantime, a first amendment to the 2nd edition is at draft amendment state including additional profiles (mainly adding xlink support) and time synchronization. A second amendment goes to the first ballot stage referred to as proposed draft amendment and defines spatial relationship description, generalized URL parameters, and other extensions. Eventually, these two amendments will be integrated in the 2nd edition which will become the MPEG-DASH 3rd edition. Also a corrigenda on the 2nd edition is currently under ballot and new contributions are still coming in, i.e., there is still a lot of interest in DASH. For your information – there will be two DASH-related sessions at Streaming Forum 2014.

On the other hand, MMT’s amendment 1 is currently under ballot and amendment 2 defines header compression and cross-layer interface. The latter has been progressed to a study document which will be further discussed at the next meeting. Interestingly, there will be a MMT developer’s day at the 109th MPEG meeting as in Japan, 4K/8K UHDTV services will be launched based on MMT specifications and in Korea and China, implementation of MMT is now under way. The developer’s day will be on July 5th (Saturday), 2014, 10:00 – 17:00 at the Sapporo Convention Center. Therefore, if you don’t know anything about MMT, the developer’s day is certainly a place to be.

Contact:

Dr. Christian Timmerer
CIO bitmovin GmbH | christian.timmerer@bitmovin.net
Alpen-Adria-Universität Klagenfurt | christian.timmerer@aau.at

What else? That is, some publicly available MPEG output documents… (Dates indicate availability and end of editing period, if applicable, using the following format YY/MM/DD):

• Text of ISO/IEC 13818-1:2013 PDAM 7 Carriage of Layered HEVC (14/05/02) 
• WD of ISO/IEC 13818-1:2013 AMD Carriage of Green Metadata (14/04/04) 
• WD of ISO/IEC 13818-1:2013 AMD Carriage of 3D Audio (14/04/04) 
• WD of ISO/IEC 13818-1:2013 AMD Carriage of additional audio profiles & levels (14/04/04) 
• Text of ISO/IEC 14496-12:2012 PDAM 4 Enhanced audio support (14/04/04) 
• TuC on sample variants, signatures and other improvements for the ISOBMFF (14/04/04) 
• Text of ISO/IEC CD 14496-22 3rd edition (14/04/04) 
• Text of ISO/IEC CD 14496-31 Video Coding for Browsers (14/04/11) 
• Text of ISO/IEC 15938-5:2005 PDAM 5 Multiple text encodings, extended classification metadata (14/04/04) 
• WD 2 of ISO/IEC 15938-6:201X (2nd edition) (14/05/09) 
• Text of ISO/IEC DIS 15938-13 Compact Descriptors for Visual Search (14/04/18) 
• Test Model 10: Compact Descriptors for Visual Search (14/05/02) 
• WD of ARAF 2nd Edition (14/04/18) 
• Use cases for ARAF 2nd Edition (14/04/18) 
• WD 5.0 MAR Reference Model (14/04/18) 
• Logistic information for the 5th JAhG MAR meeting (14/04/04) 
• Text of ISO/IEC CD 23000-15 Multimedia Preservation Application Format (14/04/18) 
• WD of Implementation Guideline of MP-AF (14/04/04) 
• Requirements for Publish/Subscribe Application Format (PSAF) (14/04/04) 
• Preliminary WD of Publish/Subscribe Application Format (14/04/04) 
• WD2 of ISO/IEC 23001-4:201X/Amd.1 Parser Instantiation from BSD (14/04/11) 
• Text of ISO/IEC 23001-8:2013/DCOR1 (14/04/18) 
• Text of ISO/IEC DIS 23001-11 Green Metadata (14/04/25) 
• Study Text of ISO/IEC 23002-4:201x/DAM2 FU and FN descriptions for HEVC (14/04/04) 
• Text of ISO/IEC 23003-4 CD, Dynamic Range Control (14/04/11) 
• MMT Developers’ Day in 109th MPEG meeting (14/04/04) 
• Results of CfP on Screen Content Coding Tools for HEVC (14/04/30) 
• Study Text of ISO/IEC 23008-2:2013/DAM3 HEVC Scalable Extensions (14/06/06) 
• HEVC RExt Test Model 7 (14/06/06) 
• Scalable HEVC (SHVC) Test Model 6 (SHM 6) (14/06/06) 
• Report on HEVC compression performance verification testing (14/04/25) 
• HEVC Screen Content Coding Test Model 1 (SCM 1) (14/04/25) 
• Study Text of ISO/IEC 23008-2:2013/PDAM4 3D Video Extensions (14/05/15) 
• Test Model 8 of 3D-HEVC and MV-HEVC (14/05/15) 
• Text of ISO/IEC 23008-3/CD, 3D audio (14/04/11) 
• Listening Test Logistics for 3D Audio Phase 2 (14/04/04) 
• Active Downmix Control (14/04/04) 
• Text of ISO/IEC PDTR 23008-13 Implementation Guidelines for MPEG Media Transport (14/05/02) 
• Text of ISO/IEC 23009-1 2nd edition DAM 1 Extended Profiles and availability time synchronization (14/04/18) 
• Text of ISO/IEC 23009-1 2nd edition PDAM 2 Spatial Relationship Description, Generalized URL parameters and other extensions (14/04/18) 
• Text of ISO/IEC PDTR 23009-3 2nd edition DASH Implementation Guidelines (14/04/18) 
• MPEG vision for Compact Descriptors for Video Analysis (CDVA) (14/04/04) 
• Plan of FTV Seminar at 109th MPEG Meeting (14/04/04) 
• Draft Requirements and Explorations for HDR /WCG Content Distribution and Storage (14/04/04) 
• Working Draft 2 of Internet Video Coding (IVC) (14/04/18) 
• Internet Video Coding Test Model (ITM) v 9.0 (14/04/18) 
• Uniform Timeline Alignment (14/04/18) 
• Plan of Seminar on Hybrid Delivery at the 110th MPEG Meeting (14/04/04) 
• WD 2 of MPEG User Description (14/04/04)

Not All Packets Are Equal

... is the title of a IEEE Internet Computing article comprising two parts.

Part 1 - Streaming Video Coding and SLA Requirements - describes the some Quality of Service (QoS) characteristics (i.e., delay, jitter, packet loss) and details some of the coding principles adopted in MPEG standards (and btw. others): subsampling, intra coding, inter coding, blocks, macroblocks, slices, frames, group of pictures (GoPs), decoding order vs. transmission order, and MPEG encapuslation within IP.

Part 2 - The Impact of Network Packet Loss on Video Quality - highlights the impact that different durations of IP packet loss have on the Quality of Experience (QoE) for IP-based video streaming services. It describe the visual impairments that result from such packet losses and present the results of testing and analysis to compare impairments for different loss durations for both MPEG-2-encoded standard and high-definition services.

Both arcticles are a very good starting point to get an overview of video coding and how to stream video data over IP-based networks. However, it mainly focuses on MPEG-2 and only mentions the current state-of-the-art codec, namely Advanced Video Coding (AVC). There are already papers available that investigate the impact of packet loss on video quality for AVC.

Nevertheless, the two papers are worth reading, written in an easy-to-read style, and also suitable for an audience not so familiar with video coding and transmission issues. Finally, it seems that there's another article planned in this series as indicated in the last sentence:

"In a future article, we hope to compare different network technology approaches for minimizing and recovering from video packet loss."

I'm looking forward to that ... (and please, bring more on AVC) -- thanks!

References

• J. Greengrass, J. Evans, A. C. Begen, "Not All Packets Are Equal, Part I: Streaming Video Coding and SLA Requirements," IEEE Internet Computing, vol. 13, no. 1, pp. 70-75, Jan./Feb. 2009
• J. Greengrass, J. Evans, A. C. Begen, "Not All Packets Are Equal, Part 2: The Impact of Network Packet Loss on Video Quality," IEEE Internet Computing, vol. 13, no. 2, pp. 74-82, Mar./Apr. 2009

STreamingDay 2008, Sep. 2, Parma, Italy

At this year’s StreamingDay – kindly hosted by University of Parma – we saw a lot of interesting presentations related to video coding (AVC, reconfigurable coding, etc.), streaming, adaptation, cross-layer optimizations, peer-to-peer, robustness, and quality monitoring (see the final program for details). With this blog entry I’d like to briefly summarize the highlights of this event.

The first session was extremely hardware-oriented where they’re aiming at efficient multimedia implementations (i.e., mainly AVC encoding but also general purpose signal processing) for the ST240 VLIW processor. The overall goal is to allow for real-time encoding of HD (1080p) content which opens the door to a punch of applications.

The second session was dedicated to streaming, adaptation, and optimization issues. Michael presented our joint work (with ST) in this session (see picture). Overall, the feedback was quite good but the overhead issue is still omnipresent. In any case, the optimized reference software implementation (note: of SVC’s bitstream extractor) provides the lower boundary of what can be practically achieved.

The last session was related P2P, robustness, and quality monitoring. The P2P presentation was a good overview of P2P systems for live video streaming. In this work the authors proposed a combination of MDC and SVC for streaming of live video in P2P systems which results in higher coding efficiency and robustness but at which costs (performance and overhead)?

Conclusion: the STreamingDay is a very nice event organized by ST and related universities and you may hear a lot about the newest trends regarding streaming, coding, and applications, also from an industry perspective... hope to see you next year.