MPEG news: a report from the 143rd meeting

The original blog post can be found at the Bitmovin Techblog and has been modified/updated here to focus on and highlight research aspects. Additionally, this version of the blog post will also be posted at ACM SIGMM Records.

MPEG News Archive

The 143rd MPEG meeting took place in person in Geneva, Switzerland. The official press release can be accessed here and includes the following details:

• MPEG finalizes the Carriage of Uncompressed Video and Images in ISOBMFF
• MPEG reaches the First Milestone for two ISOBMFF Enhancements
• MPEG ratifies Third Editions of VVC and VSEI
• MPEG reaches the First Milestone of AVC (11th Edition) and HEVC Amendment
• MPEG Genomic Coding extended to support Joint Structured Storage and Transport of Sequencing Data, Annotation Data, and Metadata
• MPEG completes Reference Software and Conformance for Geometry-based Point Cloud Compression

We have adjusted the press release to suit the audience here and emphasized research on video technologies. This blog post centers around ISOBMFF and video codecs. As always, I will conclude with an update on MPEG-DASH.

ISOBMFF Enhancements

The ISO Base Media File Format (ISOBMFF) supports the carriage of a wide range of media data such as video, audio, point clouds, haptics, etc., which has now been further extended to uncompressed videos and images.

ISO/IEC 23001-17 – Carriage of uncompressed video and images in ISOBMFF – specifies how uncompressed 2D image and video data is carried in files that comply with the ISOBMFF family of standards. This encompasses a range of data types, including monochromatic and color data, transparency (alpha) information, and depth information. The standard enables the industry to effectively exchange uncompressed video and image data while utilizing all additional information provided by the ISOBMFF, such as timing, color space, and sample aspect ratio for interoperable interpretation and/or display of uncompressed video and image data.

ISO/IEC 14496-15 (based on ISOBMFF), provides the basis for "network abstraction layer (NAL) unit structured video coding formats" such as AVC, HEVC, and VVC. The current version is the 6th edition, which has been amended to support neural-network post-filter supplemental enhancement information (SEI) messages. This amendment defines the carriage of the neural-network post-filter characteristics (NNPFC) SEI messages and the neural-network post-filter activation (NNPFA) SEI messages to enable the delivery of (i) a base post-processing filter and (ii) a series of neural network updates synchronized with the input video pictures/frames.

Research aspects: While the former, the carriage of uncompressed video and images in ISOBMFF, seems to be something obvious to be supported within a file format, the latter enables to use neural network-based post-processing filters to enhance video quality after the decoding process, which is an active field of research. The current extensions with the file format provide a baseline for the evaluation (cf. also next section). 

Video Codec Enhancements

MPEG finalized the specifications of the third editions of the Versatile Video Coding (VVC, ISO/IEC 23090-3) and the Versatile Supplemental Enhancement Information (VSEI, ISO/IEC 23002-7) standards. Additionally, MPEG issued the Committee Draft (CD) text of the eleventh edition of the Advanced Video Coding (AVC, ISO/IEC 14496-10) standard and the Committee Draft Amendment (CDAM) text on top of the High Efficiency Video Coding standard (HEVC, ISO/IEC 23008-2).

These SEI messages include two systems-related SEI messages, (a) one for signaling of green metadata as specified in ISO/IEC 23001-11 and (b) the other for signaling of an alternative video decoding interface for immersive media as specified in ISO/IEC 23090-13. Furthermore, the neural network post-filter characteristics SEI message and the neural-network post-processing filter activation SEI message have been added to AVC, HEVC, and VVC.

The two SEI messages for describing and activating post-filters using neural network technology in video bitstreams could, for example, be used for reducing coding noise, spatial and temporal upsampling (i.e., super-resolution and frame interpolation), color improvement, or general denoising of the decoder output. The description of the neural network architecture itself is based on MPEG’s neural network representation standard (ISO/IEC 15938 17). As results from an exploration experiment have shown, neural network-based post-filters can deliver better results than conventional filtering methods. Processes for invoking these new post-filters have already been tested in a software framework and will be made available in an upcoming version of the VVC reference software (ISO/IEC 23090-16).

Research aspects: SEI messages for neural network post-filters (NNPF) for AVC, HEVC, and VVC, including systems supports within the ISOBMFF, is a powerful tool(box) for interoperable visual quality enhancements at the client. This tool(box) will (i) allow for Quality of Experience (QoE) assessments and (ii) enable the analysis thereof across codecs once integrated within the corresponding reference software. 

MPEG-DASH Updates

The current status of MPEG-DASH is depicted in the figure below:

The latest edition of MPEG-DASH is the 5th edition (ISO/IEC 23009-1:2022) which is publicly/freely available here. There are currently three amendments under development:

• ISO/IEC 23009-1:2022 Amendment 1: Preroll, nonlinear playback, and other extensions. This amendment has been ratified already and is currently being integrated into the 5th edition of part 1 of the MPEG-DASH specification.
• ISO/IEC 23009-1:2022 Amendment 2: EDRAP streaming and other extensions. EDRAP stands for Extended Dependent Random Access Point and at this meeting the Draft Amendment (DAM) has been approved. EDRAP increases the coding efficiency for random access and has been adopted within VVC.
• ISO/IEC 23009-1:2022 Amendment 3: Segment sequences for random access and switching. This amendment is at Committee Draft Amendment (CDAM) stage, the first milestone of the formal standardization process. This amendment aims at improving tune-in time for low latency streaming.

Additionally, MPEG Technologies under Consideration (TuC) comprises a few new work items, such as content selection and adaptation logic based on device orientation and signaling of haptics data within DASH.

Finally, part 9 of MPEG-DASH -- redundant encoding and packaging for segmented live media (REAP) -- has been promoted to Draft International Standard (DIS). It is expected to be finalized in the upcoming meetings.

Research aspects: Random access has been extensively evaluated in the context of video coding but not (low latency) streaming. Additionally, the TuC item related to content selection and adaptation logic based on device orientation raises QoE issues to be further explored.

The 144th MPEG meeting will be held in Hannover from October 16-20, 2023. Click here for more information about MPEG meetings and their developments.

University assistant predoctoral (all genders welcome) (in German: Universitätsassistent:in)

Department of Information Technology

Scientific Staff  | Part time
Application deadline:  9 August 2023
Reference code: 403/23

Announcement

The University of Klagenfurt, with approximately 1,500 employees and over 12,000 students, is located in the Alps-Adriatic region and consistently achieves excellent placements in rankings. The motto “per aspera ad astra” underscores our firm commitment to the pursuit of excellence in all activities in research, teaching and university management. The principles of equality, diversity, health, sustainability and compatibility of work and family life serve as the foundation for our work at the university.

The University of Klagenfurt is pleased to announce the following open position at the Department of Information Technology at the Faculty of Technical Sciences with a negotiable starting date, commencing on November 13, 2023 at the latest:

University assistant predoctoral (all genders welcome) (in German: Universitätsassistent:in)

within the Ada Lovelace Programme (Developing a Virtual-Reality Game that Fosters Empathy and Perspective-Taking). The successful candidate is expected to complete a dissertation project pertaining to the development of a virtual-reality game that fosters empathy and perspective-taking.

Level of employment: 75 % (30 hours/week)

Minimum salary: € 34,412 per annum (gross); Classification according to collective agreement: B1

Contract duration: 4 years

Application deadline: by August 9, 2023

Reference code: 403/23

Area of responsibility

• Autonomous scientific work including the publication of research articles in the fields of game studies and game engineering and psychology
• Conducting independent scientific research with the aim to submit a dissertation and acquire a doctoral degree in technical sciences
• Teaching exercises and lab courses (e.g., in the game studies and engineering Master’s programme)
• Participating in research projects of the department, especially within the Ada Lovelace Programme (Developing a Virtual-Reality Game that Fosters Empathy and Perspective-Taking)
• Mentoring students
• Assisting in public relations activities, science to public communication, and extra-curricular events of the department and the faculty

Requirements

• Completed Diploma or Master’s degree from a recognized university in the field of Game Studies and Game Engineering or Computer Science. The completion of this degree must be fulfilled no later than two weeks before the starting date; hence, the last possible deadline for meeting this requirement is October 30, 2023
• Strong background in one or more of the following fields: (video) games research, empirical research methods, design and creation of video games
• Fluent in written and spoken English
• Very good scientific communication and dissemination skills (scientific writing and oral presentations)
• Programming experience in video game development

Desired skills

• Experience in interdisciplinary research projects, ideally in the social sciences, as the project involves empirical research on relevant experiences
• Social competences, reliability, and teamwork competency
• Ability to communicate with researchers of different academic backgrounds
• Scientific curiosity and enthusiasm for research in video games
• First experiences with scientific publications

Additional information

The doctoral student will be co-supervised by Mathias Lux, Judith Glück, and Felix Schniz.

Our offer:

This position serves the purposes of the vocational and scientific education of graduates of Master’s or Diploma degree programmes and sets the global of completing a Doctoral degree / a Ph.D. in Technical Sciences. Applications by persons who have already completed a subject-specific Doctoral degree or a subject-relevant Ph.D. programme therefore cannot be considered.

The employment contract is concluded for the position as university assistant (predoctoral) and stipulates a starting salary of € 2,458 gross per month (14 times a year; previous experience deemed relevant to the job can be recognised in accordance with the collective agreement).

The university of Klagenfurt is a young and vibrant campus university hosting 12,000 students. It is located in the southern part of Austria, a region with outstanding quality of life at the intersection of Alpine and Mediterranean culture.

The University of Klagenfurt also offers:

• Personal and professional advanced training courses, management and career coaching
• Numerous attractive additional benefits, see also https://jobs.aau.at/en/the-university-as-employer/
• Diversity- and family-friendly university culture
• The opportunity to live and work in the attractive Alps-Adriatic region with a wide range of leisure activities in the spheres of culture, nature and sports

The application:

If you are interested in this position, please apply in English providing the following documents:

• Letter of application
• Curriculum vitae (with clear information about the degrees including date/place/grade, the experience acquired, the thesis title, the list of publications if any, and any other relevant information)
• Copy of the degree certificates and transcripts of the courses
• Any certificates that can prove the fulfilment of the required and additional qualifications listed above (e. g., the submission of the final thesis/dissertation if required by the degree programme, copy of publications programming skills certificates, language skills certificates etc.)
• Final thesis or other study-related written work (like seminar reports) or excerpts thereof
• If an applicant has not received the Diploma or Master’s degree by the application deadline, the applicant should provide a declaration, written either by a supervisor or by the candidate themselves, on the feasibility of finishing the Diploma or Master’s degree by October 30, 2023 at the latest.

To apply, please select the position with the reference code 403/23 in the category “Scientific Staff” using the link “Apply for this position”.

Candidates must furnish proof that they meet the required qualifications by October 30, 2023 at the latest.

For further information on this specific vacancy, please contact Assoc.-Prof. DI Dr. Mathias Lux (mathias.lux@aau.at). General information about the university as an employer can be found at https://jobs.aau.at/en/the-university-as-employer/. At the University of Klagenfurt, recruitment and staff matters are accompanied not only by the authority responsible for the recruitment procedure but also by the Equal Opportunities Working Group and, if necessary, by the Representative for Disabled Persons.

The university of Klagenfurt aims to increase the proportion of women and therefore specifically invites qualified women to apply for the position. Where the qualification is equivalent, women will be given preferential consideration.

People with disabilities or chronic diseases, who fulfil the requirements, are particularly encouraged to apply.

Travel and accommodation costs incurred during the application process will not be refunded.

Translations into other languages shall serve informational purposes only. Solely the version advertised in the University Bulletin (Mitteilungsblatt) shall be legally binding.

Apply for this position

Universitätsassistent:in

Institut für Informationstechnologie

Wissenschaftliches Universitätspersonal  | Teilzeit
Bewerbungsfrist:  9. August 2023
Kennung: 405/23

Ausschreibung

Die Universität Klagenfurt ist mit rund 1 500 Mitarbeitenden und über 12 000 Studierenden im Alpen-Adria-Raum angesiedelt und erreicht in Rankings regelmäßig exzellente Platzierungen. Das Motto „per aspera ad astra“ bringt den Anspruch des konsequenten Strebens nach Spitzenleistungen bei allen Tätigkeiten in Forschung, Lehre und Hochschulmanagement zum Ausdruck. Die Prinzipien der Gleichstellung, der Diversität, der Gesundheit, der Nachhaltigkeit und der Vereinbarkeit von Beruf und Familie bilden die Grundlage für das Arbeiten an der Universität.

Am Institut für Informationstechnologie an der Fakultät für Technische Wissenschaften wird voraussichtlich mit 1. Oktober 2023 folgende Stelle besetzt:

Universitätsassistent:in

Beschäftigungsausmaß: 75 % (30 Wo.std.); mit der Möglichkeit zur Erhöhung des Beschäftigungsausmaßes auf 100 % (40 Wo.std.) aus Drittmitteln

Mindestentgelt: € 34.412,-- brutto jährlich; Einstufung nach Uni-KV: B1

Befristung: 4 Jahre

Bewerbungsfrist: bis 9. August 2023

Kennung: 405/23

Aufgabenbereich

• Selbstständiges wissenschaftliches Arbeiten, einschließlich der Veröffentlichung von Forschungsartikeln auf den Gebieten Game Studies, Game Engineering und Informatik
• Durchführung unabhängiger wissenschaftlicher Forschung mit dem Ziel, eine Dissertation einzureichen und einen Doktorgrad der technischen Wissenschaften zu erlangen
• Durchführung von Übungen und Laborpraktika auf Deutsch (z.B.: Rechnerorganisation, Betriebssysteme, Computernetzwerke)
• Aktive Teilnahme an Forschungsprojekten am Institut
• Betreuung von Studierenden
• Unterstützung bei Öffentlichkeitsarbeit, Wissenschaftskommunikation und Veranstaltungen des Instituts und der Fakultät

Voraussetzungen

• Abgeschlossenes Diplom- oder Masterstudium an einer anerkannten Universität im Bereich Game Studies und Game Engineering oder Informatik. Der Abschluss dieses Studiums muss spätestens zwei Wochen vor dem geplanten Dienstantritt erfolgen; daher ist die letztmögliche Frist zur Erfüllung dieser Anforderung der 17. September 2023
• Solide Kenntnisse in einem oder mehreren der folgenden Bereiche: (Video-)Spielforschung, empirische Forschungsmethoden, Gestaltung und Erstellung von Videospielen
• Sehr gute Beherrschung der deutschen und englischen Sprache in Wort und Schrift
• Ausgezeichnete wissenschaftliche Kommunikationsfähigkeiten (wissenschaftliches Schreiben und wissenschaftliche Vorträge)
• Programmiererfahrung, insbesondere im Bereich der Entwicklung von Videospielen

Erwünscht sind

• Erfahrung in interdisziplinären Forschungsprojekten
• Soziale Kompetenz, Zuverlässigkeit und Teamfähigkeit
• Fähigkeit zur Kommunikation mit Forscher:innen aus unterschiedlichen akademischen Bereichen
• Bereitschaft aktiv an Forschung, Lehre und Verwaltungsaufgaben teilzunehmen
• Erste wissenschaftliche Veröffentlichungen

Zusatzinformation

Das Angebot:

Der Dienstvertrag wird mit einem Einstiegsentgelt von mtl. € 2.458,-- brutto (14x jährlich; eine Anrechnung tätigkeitsspezifischer Vorerfahrung gemäß Kollektivvertrag ist möglich) abgeschlossen.

Zudem bietet die Universität Klagenfurt:

• Persönliche und berufliche Weiterbildungsangebote, Führungskräfte- und Karrierecoaching
• Zahlreiche attraktive Zusatzleistungen, siehe dazu jobs.aau.at/arbeitgeber-universitaet-klagenfurt/
• Diversitäts- und familienfreundliche Universitätskultur
• Leben und arbeiten in der attraktiven Alpen-Adria-Region mit vielfältigen Freizeitmöglichkeiten in den Bereichen Kultur, Natur & Sport

Die Bewerbung:

Bei Interesse bewerben Sie sich in deutscher oder englischer Sprache mit den üblichen Unterlagen:

• Bewerbungsschreiben
• Lebenslauf
• Zeugnisse / Bestätigungen
• Konzept eines (möglichen) Dissertationsprojekts (maximal 1 Seite)

Diese Stelle dient der fachlichen und wissenschaftlichen Bildung von Absolvent:innen eines Master- bzw. Diplomstudiums mit dem Ziel des Abschlusses eines Doktoratsstudiums der technischen Wissenschaften (im Doktoratsprogramm Informatik). Bewerbungen von Personen, die bereits über ein facheinschlägiges Doktorat bzw. einen facheinschlägigen Ph.D. verfügen, können daher nicht berücksichtigt werden.

Bewerbungen sind ausschließlich bei der Stelle mit der Kennung 405/23 in der Rubrik „Wissenschaftliches Universitätspersonal“ über den Link „Für diese Stelle bewerben“ am Ende der Ausschreibung möglich.

Die erforderlichen Nachweise für die Einstellung müssen bis spätestens 17. September 2023 vorliegen.

Nähere Auskünfte zur konkreten Stellenausschreibung erteilen Assoc. Prof. DI Dr. Mathias Lux (mathias.lux@aau.at) oder Univ.-Prof. DI Dr. Christian Timmerer (christian.timmerer@aau.at). Allgemeine Informationen zur Universität als Arbeitgeberin finden sich unter www.aau.at/jobs/information. Die Personalverfahren werden an der Universität Klagenfurt neben der zuständigen ausschreibenden Stelle auch vom Arbeitskreis für Gleichbehandlungsfragen und ggfs. von der Behindertenvertretung begleitet.

Die Universität Klagenfurt strebt eine Erhöhung des Frauenanteils an und fordert daher qualifizierte Frauen ausdrücklich zur Bewerbung auf. Bei gleicher Qualifikation werden Frauen vorrangig aufgenommen.

Menschen mit Behinderungen oder chronischen Erkrankungen, die die geforderten Qualifikationskriterien erfüllen, werden ausdrücklich zur Bewerbung aufgefordert.

Es besteht kein Anspruch auf Abgeltung von Reise- und Aufenthaltskosten, die aus Anlass des Aufnahmeverfahrens entstehen.

Für diese Stelle bewerben

DASH-IF awarded Excellence in DASH award at ACM MMSys 2023

Vancouver, June 9, 2023

The DASH Industry Forum Excellence in DASH Award at ACM MMSys 2023 acknowledges papers substantially addressing MPEG-DASH as the presentation format and are selected for presentation at ACM MMSys 2023. Preference is given to practical enhancements and developments which can sustain future commercial usefulness of DASH. The DASH format used should conform to the DASH-IF Interoperability Points as defined by http://dashif.org/guidelines/. It is a financial prize as follows: first place – €750; second place – €500; and third place – €250. The winners are chosen by a DASH Industry Forum-appointed committee, and the results are final.

This year's award goes to the following papers:

First Place

Latency Target based Analysis of the DASH.js Player
by
Piers O’Hanlon and Adil Aslam

Abstract: We analyse the low latency performance of the three Adaptive Bitrate (ABR) algorithms in the dash.js Dynamic Adaptive Streaming over HTTP (DASH) player with respect to a range of latency targets and configuration options. We perform experiments on our DASH Testbed which allows for testing with a range of real world derived network profiles. Our experiments enable a better understanding of how latency targets affect quality of experience (QoE), and how well the different algorithms adhere to their targets. We find that with dash.js v4.5.0 the default Dynamic algorithm achieves the best overall QoE. We show that whilst the other algorithms can achieve higher video quality at lower latencies, they do so only at the expense of increased stalling. We analyse the poor performance of L2A-LL in our tests and develop modifications which demonstrate significant improvements. We also highlight how some low latency configuration settings can be detrimental to performance.

Second Place

Cross-layer Network Bandwidth Estimation for Low-latency Live ABR Streaming
by
Chinmaey Shende, Cheonjin Park, Subhabrata Sen, Bing Wang

Abstract: Low-latency live (LLL) adaptive bitrate (ABR) streaming relies critically on accurate bandwidth estimation to react to dynamic network conditions. While existing studies have proposed bandwidth estimation techniques for LLL streaming, these approaches are at the application level, and their accuracy is limited by the distorted timing information observed at the application level. In this paper, we propose a novel cross-layer approach that uses coarse-grained application-level semantics and fine-grained kernel-level packet capture to obtain accurate bandwidth estimation. We incorporate this technique in three popular open-source ABR players and show that it provides significantly more accurate bandwidth estimation than the state-of-the-art application-level approaches. In addition, the more accurate bandwidth estimation leads to better bandwidth prediction, which we show can lead to significantly better quality of experience (QoE) for end users.

Third Place

TASQ: Temporal Adaptive Streaming over QUIC
by
Akram Ansari, Yang Liu, Mea Wang, Emir Halepovic

Abstract: Traditional Adaptive BitRate (ABR) streaming faces a challenge of providing smooth experience under highly variable network conditions, especially when low latency is required. Effective adaptation techniques exist for deep-buffer scenarios, such as streaming long-form Video-on-Demand content, but remain elusive for short-form or low-latency cases, when even a short segment may be delivered too late and cause a stall. Recently proposed temporal adaptation aims to mitigate this problem by being robust to losing a part of the video segment, essentially dropping the tail of the segment intentionally to avoid the stall. In this paper, we analyze this approach in the context of a recently adopted codec AV1 and find that it does not always provide the promised benefits. We investigate the root causes and find that a combination of codec efficiency and TCP behavior can defeat the benefits of temporal adaptation. We develop a solution based on QUIC, and present the results showing that the benefits of temporal adaptation that still apply to AV1, including reduced stall time up to 65% compared to the original TCP-based approach. In addition, we present a novel way to use the stream management features of QUIC to benefit Quality-of-Experience (QoE) and reduce wasted data in video streaming.

The DASH-IF congratulates all winners and hopes to see you next year at ACM MMSys 2024.

DASH-IF awarded Best PhD Dissertation on Algorithms and Protocols for Adaptive Content Delivery over the Internet

Vancouver, June 9, 2023

The DASH Industry Forum (DASH-IF) is set to honor the most outstanding PhD dissertation on Algorithms and Protocols for Adaptive Content Delivery over the Internet during the ACM Multimedia Systems 2023. The winner of the award will be selected by a committee composed of DASH-IF members. All dissertations that fall within the scope and have been successfully defended in 2021, 2022 or 2023 (before the May 1, 2023 submission deadline) are qualified for nomination. The recipient of the award will receive 750 USD.

And the winner is

Dr. Mehmet N. Akcay
for his dissertation on
Server and Client-side Algorithms for Enhancing Adaptive Streaming

Abstract: HTTP adaptive video streaming is a technique widely used on the internet today to stream live and on-demand content. Server and client-side algorithms play an important role in improving user experience in terms of metrics such as latency, rebufferings, and rendering quality. After explaining the commonly used metrics, the author analyzed four main aspects of video streaming (i) bandwidth prediction accuracy, (ii) utilization of playback speed, (iii) adaptive streaming for content-aware-encoded videos, and (iv) head motion awareness for 360-degree videos. 360-degree video streaming requires much higher bandwidth compared to conventional video streaming. The author demonstrates that most of the algorithmic improvements achieved for video streaming can also be applied to Viewport Dependent Streaming (VDS) for 360-degree videos. It is also important that in 360-degree video streaming, we have a Head Mounted Display (HMD) device that is capable of pointing the viewport orientation of the user. We also investigate and improve the rate-adaptation algorithms for 360-degree videos by developing several new algorithms using the HMD. The new algorithms proposed in this thesis are Low-on-Latency (LoL), Low- on-Latency+ (LoL+), Bang-on-Bandwidth (BoB), Size-aware Rate Adaptation (SARA), Content-aware Playback Speed Control (CAPSC), Head-motion-aware Viewport Margins (HMAVM). We evaluate the proposed new algorithms using the objective metrics discussed in detail and show significant contributions for these new algorithms including up to 91% decrease in rebuffering duration for on-demand streaming, a 61.9% decrease in rebuffering duration, and an 8.1% decrease in latency compared to L2A for low-latency live streaming, 81.3% bandwidth prediction accuracy for interactive streaming, lastly 20% improvement in viewport quality and 50% reduction in motion- to-high-quality delay for 360-degree video streaming.

The DASH-IF congratulates the winner and hopes to see you next year at ACM MMSys 2024.

Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming

 2022 Picture Coding Symposium (PCS)

December 7-9, 2022 | San Jose, CA, USA

Conference Website
[PDF][Slides]

Vignesh V Menon (Alpen-Adria-Universität Klagenfurt),  Hadi Amirpour (Alpen-Adria-Universität Klagenfurt), Prajit T Rajendran (Universite Paris-Saclay, France), Mohammad Ghanbari (School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK), and Christian Timmerer (Alpen-Adria-Universität Klagenfurt)

Abstract: In live streaming applications, a fixed set of bitrate-resolution pairs (known as bitrate ladder) is generally used to avoid additional pre-processing run-time to analyze the complexity of every video content and determine the optimized bitrate ladder. Furthermore, live encoders use the fastest available preset for encoding to ensure the minimum possible latency in streaming. For live encoders, it is expected that the encoding speed is equal to the video framerate. However, an optimized encoding preset may result in (i) increased Quality of Experience (QoE) and (ii) improved CPU utilization while encoding. In this light, this paper introduces a Content-Adaptive encoder Preset prediction Scheme (CAPS) for adaptive live video streaming applications. In this scheme, the encoder preset is determined using Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment, the number of CPU threads allocated for each encoding instance, and the target encoding speed. Experimental results show that CAPS yields an overall quality improvement of 0.83 dB PSNR and 3.81 VMAF with the same bitrate, compared to the fastest preset encoding of the HTTP Live Streaming (HLS) bitrate ladder using x265 HEVC open-source encoder. This is achieved by maintaining the desired encoding speed and reducing CPU idle time.

CAPS_Presentation.pdf from Vignesh V Menon

Acknowledgements: The financial support of the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the Christian Doppler Research Association, is gratefully acknowledged. Christian Doppler Laboratory ATHENA: https://athena.itec.aau.at/.

IEEE TIP: Advanced Scalability for Light Field Image Coding

Advanced Scalability for Light Field Image Coding

IEEE Transactions on Image Processing (TIP)

Journal Website

[PDF]

Hadi Amirpour (Alpen-Adria-Universität Klagenfurt, Austria), Christine Guillemot (INRIA, France), Mohammad Ghanbari (University of Essex, UK), and Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Austria)

Abstract: Light field imaging, which captures both spatial and angular information, improves user immersion by enabling post-capture actions, such as refocusing and changing view perspective. However, light fields represent very large volumes of data with a lot of redundancy that coding methods try to remove. State-of-the-art coding methods indeed usually focus on improving compression efficiency and overlook other important features in light field compression such as scalability. In this paper, we propose a novel light field image compression method that enables (i) viewport scalability, (ii) quality scalability, (iii) spatial scalability, (iv) random access, and (v) uniform quality distribution among viewports, while keeping compression efficiency high. To this end, light fields in each spatial resolution are divided into sequential viewport layers, and viewports in each layer are encoded using the previously encoded viewports. In each viewport layer, the available viewports are used to synthesize intermediate viewports using a video interpolation deep learning network. The synthesized views are used as virtual reference images to enhance the quality of intermediate views. An image super-resolution method is applied to improve the quality of the lower spatial resolution layer. The super-resolved images are also used as virtual reference images to improve the quality of the higher spatial resolution layer.
The proposed structure also improves the flexibility of light field streaming, provides random access to the viewports, and increases error resiliency. The experimental results demonstrate that the proposed method achieves a high compression efficiency and it can adapt to the display type, transmission channel, network condition, processing power, and user needs.

Keywords—Light field, compression, scalability, random access, deep learning.

Acknowledgements: The financial support of the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the Christian Doppler Research Association, is gratefully acknowledged. Christian Doppler Laboratory ATHENA: https://athena.itec.aau.at/.