MPEG news: a report from the 93rd meeting, Geneva, Switzerland

Here comes a longer version of my "less than 140 chars summary" I've posted to Twitter :-)

Table of contents:

1. MPEG-V reached Final Draft International Standard (FDIS) after three years of development.
2. HTTP streaming of MPEG Media has been evaluated to Dynamic Adaptive Streaming over HTTP (DASH) working draft using 3GPP's Adaptation HTTP Streaming (AHS) as starting point.
3. A Call for Proposals for MPEG Media Transport (MMT) has been issued which will be evaluated in January 2011.

1. MPEG-V reached Final Draft International Standard (FDIS) after three years of development

MPEG-V parts (ISO/IEC 23005): see also previous posts related to MPEG-V

• ISO/IEC 23005-1: Architecture. This part is more like a technical reports describing the system architecture, standardization areas, and possible use cases as well as concrete instantiations of the architecture for various application domains.
• ISO/IEC 23005-2: Control information. This part provides description formats for describing device capabilities, user preferences, and device commands required in the context of the Representation of Sensory Effects (RoSE) framework, a possible instantiation of the MPEG-V architecture.
• ISO/IEC 23005-3: Sensory information. Together with the description formats defined in part 2 of MPEG-V it allows for a complete instantiation of the RoSE framework. In particular, it defines description formats enabling the representation of sensory effects (e.g., wind, vibration, explosion, etc.) that can be rendered on specific devices (e.g., fans, vibrating chairs, add'l light sources, etc.), hence, increasing the overall user experience.
• ISO/IEC 23005-4: Virtual world object characteristics. Part 4 of MPEG-V provides means to describe the characteristics of virtual world objects following an XML notation (btw. XML is also used by the other parts) that shall enable the interoperable exchange of such objects among virtual worlds of different kinds.
• ISO/IEC 23005-5: Data formats for interaction devices. N/A
• ISO/IEC 23005-6: Common types and tools. Defines basic data types and tools used by the other parts of MPEG-V.
• ISO/IEC 23005-7: Conformance and reference software (currently at FCD level). This part comprises the reference software as MPEG Extensible Middleware (MXM) engines and provides means for conformance testing.

Details concerning MPEG-V are discussed here.

2. Dynamic Adaptive Streaming over HTTP (DASH)

I had the honor to chair the evaluation of responses submitted to the HTTP Streaming of MPEG Media AhG. We had 15 submissions from 20 organizations (companies/institutions/universities) and the 3GPP's Adaptive HTTP Streaming has been selected as a starting point. In particular, MPEG will standardize a Manifest File (MF), a Delivery Format (DF), and means for easy conversion from/to existing File Formats (FF) and Transport Streams (TS) as depicted in the presentation below.

More details will be provided once the MPEG DASH standard has become more mature. Details concerning DASH are discussed here.

Call for Proposals for MPEG Media Transport (MMT)

A Call for Proposals for MPEG Media Transport (MMT) has been issued which will be evaluated in January 2011. All details can be found here and are not repeated here. Details concerning MMT are discussed here.

WIAMIS'10: Increasing the User Experience of Multimedia Presentations with Sensory Effects

Here comes the abstract of our paper that has been accepted for publication at WIAMIS'10. Please join us there in case you'd like to get in touch we us!

Markus Waltl, Christian Timmerer, Hermann Hellwagner, Increasing the User Experience of Multimedia Presentations with Sensory Effects, accepted for publication at 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.

Abstract:
The term Universal Multimedia Experience (UME) has gained momentum and is well recognized within the research community. As this approach puts the user into the center stage, additional complexity is added to the overall quality assessment problem which calls for a scientific framework to capture, measure, quantify, judge, and explain the user experience. In previous work we have proposed the annotation of multimedia content with sensory effect metadata that can be used to stimulate also other senses than vision or audition. In this paper we report first results obtained from subjective tests in the area of sensory effects attached to traditional multimedia presentations such as movies that shall lead to an enhanced, unique, and worthwhile user experience.

See here the list of accepted papers at WIAMIS'10.

Overview of Selected Current MPEG Activities

--this covers a report from the 91st MPEG meeting in Kyoto, Japan

Previously, I've always provided a written report but this time it comes in form of a presentation (slideshow) - enjoy!

View more presentations from Klagenfurt University.

MPEG news: a report from the 87th meeting in Lausanne, Switzerland

MPEG’s high-performance video coding (HVC) standard is evolving and currently targets mobile devices, IPTV, and Ultra-HD. However, the trade-off between coding efficiency and codec complexity is still driving the thresholds which has been set at this meeting, well, at least initially. For low-complexity HVC is seeking for 25% gain in coding efficiency and for full-/increased-complexity the threshold is set to 50% gain in coding efficiency (cf. goal for AVC standardization). The application scenarios range from shared Ultra-HD to personalized experiences. The latter is targeting a viewing distance of 0.5*h (i.e., 50cm) and for personal use only. A vision document (N10361) has been issued at this meeting and interested parties are requested to join the Ad-hoc Group (AhG) for further discussions (N10371). One open issue with HVC is whether and how audio will adapt to these new developments, probably with high-performance audio coding (HAC) or can HE-AAC already complement HVC? Anyway, short viewing distance ultimately leads to short listening distances and a wide angle for the sound stage...

3D video coding (3DVC) has already a history within MPEG. A vision document has been produced here stating that MPEG’s version of 3D video shall be compatible with existing standards, mono/stereo devices, and existing/planned infrastructure. While the former seems to be very clear, i.e., backwards compatibility to AVC is desirable in the same way as it was for SVC, the latter needs some more discussions. How can one be compatible with planned infrastructures?

In the area of advanced IPTV terminal (AIT) we had a productive meeting with ITU-T SG16 Q.13. The objectives look very promising and aim to increase the user experience thanks to the usage of latest media coding, transport, and processing technologies. An AhG has been set up to discuss the details. Again, AhGs are open to the public and everyone can join and contribute.

The information exchange with virtual worlds (MPEG-V) has merged with the representation of sensory effects (RoSE) and will be now jointly developed under the umbrella of MPEG-V. However, it was sought that ‘information exchange with virtual worlds’ is no longer appropriate and they’re seeking for a new name. Any inputs are welcome and are discussed within the AhG that is open to everybody. Thus, join this exciting activity now! In its current working draft the following sensory effects are defined which shall create an enhanced, immersive user experience: various light effects, temperature, wind, vibration, water sprayer, perfumer/scent, fog, window blind/shadow, sound, and color correction.

Finally, MPEG got another Emmy, the "Technology and Engineering Emmy Award 2007-2008" for MPEG-4 AVC from the National Academy of Television Arts & Sciences (NATAS). It seems that success within MPEG is from now on measured in terms of number of awards received ;-) And last but not least, if you’ve been ever interested in the MPEG vision, you may find it here.

Technik LIVE @ Klagenfurt University

Dec 19, 2008: We had a couple of workshops here at Klagenfurt University where we invited local schools - in and around Klagenfurt. Our group was also there with workshop demonstrating the amBX technology that is used in an internal project here and also lead to a couple of MPEG contributions.

A video of this event is available on YouTube. Note, it's in German only...

Working Draft of Representation of Sensory Effects

The Working Draft of Representation of Sensory Effects is available and you may also find an updated presentation thereof on SlideShare.

See also previous posts for details.

MPEG news: a report from the 85th meeting in Busan, Korea

A lot of interesting things happened here and I'd like to report on three topics:

• MPEG RoSE
• Advanced IPTV Terminal
• High-performance Video Coding
  

MPEG RoSE: At this meeting we've issued the second version of the WD which will be publicly available and, thus, I can provide a more detailed overview here. I've also updated my slides from the last meeting which now can be found on SlideShare. The aim of RoSE to extend the traditional A/V content consumption to the dimension of sensory effects which are referred to as "an effect to augment perception by stimulating human senses in a particular scene of a multimedia application". With that definition in mind, sensory effects are composed by following the structure of the Sensory Effect Description Language (SEDL) and making use of terms (actually, effect types) of the Sensory Effect Vocabulary (SEV).

Sensory Effect Description Language (SEDL): Provides basic building blocks (declaration, effects, group of effects, reference to effects, parameters), common attributes (timing, priority, intensity, etc.), and data types (void at the moment) for constructing/authoring sensory effect metadata which is associated to A/V content.

Sensory Effect Vocabulary (SEV): Provides a clear set of extensible effect types which are currently comprising effects for color (illumination), temperature (°C), wind (Beaufort), and vibration (Richter).

It is foreseen that the sensory effect metadata which comes along with the A/V content is translated (or mapped or adapted) - by a module (hw/sw) called RoSE engine - to commands that are understood by so-called sensory devices with certain capabilities. Both commands and capabilities are also within the scope of standardization. Hence, it should be possible to consume the A/V content timely synchronized with its effects for an increased user experience. Furthermore, user preferences might also affect this translation/mapping/adaptation process.

Advanced IPTV Terminal: This activity is about to define a terminal suitable for IPTV scenarios jointly with ITU which may result in a similar construction as the JVT for video coding standardization. However, nothing has been fixed yet and interested parties are invited to join the discussions via the corresponding Ad-hoc Group (AhG) [subscribe]with the following mandates:

1. Look after the process of establishing the joint project with ITU-T so that SG16 may presented with a matured proposal for a new standard for Advanced IPTV Terminal
2. Gather use case scenarios and requirements for Advanced IPTV Terminal
3. Conduct collaboration with ITU-T Q.13/16 to prepare a joint meeting

High-performance Video Coding: I've already reported on that in previous blog posts, see part one and two for details.

Finally, MPEG got an Emmy! ;-)

MPEG news: a report from the 85th meeting in Hannover, Germany

The 85th MPEG meeting was held in Hannover, Germany and a few topics discussed at this meeting I'd like to briefly review here.

Representation of Sensory Effects

A working draft of the MPEG Representation of Sensory Effects (RoSE) has been issued and I've given an overview at the Friday plenary. Interested readers might also have a look at the overview paper issued at the last meeting.

Multimedia Extensible Middleware

The Multimedia Extensible Middleware (MXM) also reached the working draft status after a very comprehensive contribution of more than 10 institutions has been submitted as a response to the CfP. It will provide an API for accessing various assets along the multimedia service delivery chain. I guess I'll provide a story soon about that...

Rich media user interface framework

media user interface framework is MPEG's approach getting closer to the actual user. At the moment four areas of interest have been defined: (1) a widget authoring format, (2) widget representation format, (3) widget/container interface, (4) widget/container server protocol. A context and objective document has been issued at this meeting and will appear soon on the MPEG Web site's exploration section.

MPEG-V

There has been quite some activity on MPEG-V, also related to RoSE. In fact, the system architecture of MPEG-V has been revised so that it maps onto the RoSE architecture. That is, now the RoSE architecture can be seen as a possible instantiation of the MPEG-V system architecture. Having said this, I think 'alea iacta est', RoSE will become part of MPEG-V. Of course, this is my personal opinion and please do not quote me on that ;-) Below you can find the architecture of MPEG-V and details will appear soon on the MPEG Web site's exploration section also.

Next Generation Video Coding

Industry is demanding for resolution, resolution, resolution beyond HD sometimes also known as Ultra High Defintion Video. Therefore, at the next meeting in Busan there will be a workshop prior (or after) the meeting where new ideas can be presented. However, for some reason it seems that we will already see a solution for that at this year's IBC which is based - now sit down - on MPEG-2.

Finally, this is the place where we had our social event - it was really very nice & thanks to the organizers of this meeting. Enjoy your holidays, you have deserved it.

Representation of Sensory Effects: Call for Proposals

-- excerpted from RoSE CfP and Requirements
-- see also my previous blog entries: 1st, 2nd

System Architecture and Terminology
The overall system architecture for the RoSE framework is depicted in the figure below and comprises Sensory Effect Metadata, Sensory Device Capabilities, Sensory Device Commands, User Sensory Preferences, and a so-called RoSE Engine which generates output data based on its input data. It is important to note that the Sensory Effect Metadata, Sensory Device Capabilities, Sensory Device Commands, and User Sensory Preferences are within the scope of standardization and, thus shall be normatively specified. On the other side, the RoSE Engine as well as Provider entities and Consumer Devices are informative and are left open for industry competition.

Provider: an entity that acts as the source of the Sensory Effect Metadata (e.g., broadcaster, …)

RoSE Engine: an entity that takes the Sensory Effect Metadata, the Sensory Device Capabilities and the User Sensory Preferences as inputs and generates Sensory Device Commands based those (e.g., a commercial product like the amBX engine).

Consumer Devices: entities that act as the sink of the Sensory Commands and act as the source of Sensory Device Capabilities. Additionally, entities that provide User Sensory Preferences are also referred to as Consumer Devices. Note: Sensory Devices are sub-set of Consumer Devices. Example: Fans, Lights, scent devices, human input devices such as a TV set with a remote control (for preferences).

Sensory Effect Metadata: it defines the description schemes and descriptors to represent Sensory Effects (e.g., XML schema, …).

Sensory Effect: a Sensory Effect is an effect to augment feeling by stimulating human sensory organs in a particular scene of a multimedia application (e.g., scent, wind, light, …).

Sensory Effect Delivery Format: a standard that defines means for transporting Sensory Effect Metadata. Note: This could be combined with the Audio/Video delivery format. Example: MPEG2-TS payload format, file format, RTP payload format, …

Sensory Device Capability: it defines the description schemes and descriptors to represent the characteristics of Sensory Devices (e.g., XML schema, ...).

Sensory Device: a Sensory Device is a device by which the corresponding Sensory Effect can be made (e.g., lights, fans, heater, fan, …).

Sensory Device Capability Delivery Format: a standard that defines means for transporting Sensory Device Capabilities (e.g., HTTP, UPnP, …).

Sensory Device Command: it defines the description schemes and descriptors to control Sensory Devices (e.g., XML schema, ...).

Sensory Device Command Delivery Format: a standard that defines means for transporting Sensory Device Commands (e.g., HTTP, UPnP, …).

User Sensory Preferences: it defines the description schemes and descriptors to represent user preferences with respect to rendering of Sensory Effects (e.g., XML schema, ...).

User Sensory Preferences Delivery Format: a standard that defines means for transporting User Sensory Preferences (e.g., HTTP, UPnP, …).

The current call focuses on Sensory Effect Metadata, Sensory Device Capabilities, Sensory Device Commands, and User Sensory Preferences.

MPEG news: a report from the 84th meeting in Archamps, France

The 84th MPEG meeting was held in Archamps, France (next to Geneva) and a few topics discussed at this meeting I'd like to briefly review here.

MPEG Representation of Sensory Effects

I've reported on that recently and this topic is maturing to a Call for Proposals (CfP) and the proposed technologies will be evaluated during the July meeting in Hannover, Germany. In this CfP MPEG is requesting technologies for the following items to be standardized:

• Sensory Effect Metadata: description schemes and descriptors representing Sensory Effects.
  
• Sensory Device Capabilities and Commands: description schemes and descriptors representing characteristics of Sensory Devices and means to control them.
  
• User Sensory Preferences: description schemes and descriptors representing user preferences with respect to rendering of sensory effects.
  

Microsoft has adopted MPEG-21 Digital Item Declaration and Identification

Microsoft has adopted MPEG-21 technology within their Interactive Media Manager that is a collaborative media management solution that extends Microsoft Office SharePoint® Server 2007 for media and entertainment companies. Interestingly, their have adopted the Digital Item Model - an abstract model expressed in EBNF - and defined their own implementation of this model using RDF/OWL. Note that MPEG's implementation of the model is called Digital Item Declaration Language (DIDL) which is based on XML Schema.

Note: also UPnP has adopted MPEG-21 DIDL but in some kind of dialect called DIDL-lite.

Multimedia Extensible Middleware (MXM)

The Multimedia Extensible Middleware (MXM) aims to define APIs for various purposes. The requirements and also a Call for Proposals (requirements) have been issued at this meeting. It's very interesting to see that it should also accommodate for peer-to-peer technologies, i.e., storage/consumption of content in a distributed environment (P2P infrastructure based, e.g., distributed hash tables).

Other issues discussed at the meeting

• Presentation of Structured Information: For this item a Call for Proposals (requirements) has been issued. One use cases and overall model is to include a presentation description into Digital Items that upon receipt of a Digital Item automatically extracts this information asset which is used to present the Digital Item.
  
• MPEG User Interface Framework: personalize, adaptable and exchangeable rich user interfaces based on the use of a presentation format (BIFS, LaSER, ...), a language for describing the personalization context (UED, CC/PP, DCO, ...), and home network protocols (DLNA, UPnP, ...).
• MPEG-V: This new project item defines interfaces between virtual worlds and between virtual worlds and the real world. An extended call for requirements has been issued.
• Multimedia Value Chain Ontology (MVCO): aims to define an ontology for the whole multimedia value chain and a Call for Proposals (requirements) has been issued. However, the main focus is on rights-related issues at this moment.

What's new with MPEG: Representation of Sensory Effects (RoSE)

This article covers a brave new topic currently being discussed within the Moving Picture Experts Group (MPEG), formally known as ISO/IEC JTC 1/SC 29/WG 11. It comprises a framework for the Representation of Sensory Effects (RoSE) and its context and objectives are described in [1]. Discussions take place via [3] and is open to everyone.

Context and Objectives

The aim of RoSE is to provide a standardized framework which shall allow for controlling sensory effects in order to increase the overall experience of the user when consuming multimedia content. A sensory effect is an effect to augment feelings by stimulating sensory organs in a particular sense of a multimedia application. Examples of sensory effects include - but are not limited to - audio (sound), visual (video), tactile (light, shading, vibration, wind, temperature, fog, water), and olfactory (scent).

Scope of Standardization

The scope of standardization can be divided into description schemes (or sometimes called tools) and systems aspects.

The former - description schemes/tools - covers three topics, namely:

• Sensory Effect Schema (SES): It defines the description schemes and descriptors to represent Sensory Effects
• Sensory Device Schema (SDS): It defines the description schemes and descriptors to represent characteristics of Sensory Devices.
• User Sensory Preferences (USP): It defines the description schemes and descriptors to represent the user preferences on the (rendering) of the sensory effects.

The later - systems aspects - covers two topics, namely:

• Storage (File Format): It defines an application format which is used to package the audiovisual contents and their associated RoSE metadata.
• Transport (Delivery Format): It defines a schema of message forms by which a part (fragment) of RoSE metadata can be delivered. The specific delivery protocols may be out of the standardization scope.

Timeline

• April 2008: "Final" Requirements and Call for Proposals
• July 2008: Evaluation of proposals and Working Draft (WD)
• October 2008: Committee Draft (CD)
• April 2009: Final Committee Draft (FCD)
• October 2009: Final Draft International Standard (FDIS)

References

[1] Context and Objectives, http://www.chiariglione.org/MPEG/working_documents/explorations/RoSE/RoSE-C&O.zip

[2] Draft Requirements for RoSE, http://www.chiariglione.org/MPEG/working_documents/explorations/RoSE/RoSE-Reqs.zip

[3] RoSE AhG Reflector, http://lists.uni-klu.ac.at/mailman/listinfo/rose