O Universal Multimedia Access, Where Art Thou? (Part III)

-by Christian Timmerer, Klagenfurt University, Austria

Preface: First I thought about writing this article for a journal or something equivalent but then I concluded to make this article available through my blog. The aim is to perform an experiment in order to determine whether it is possible (a) to get direct feedback through comments and (b) to be referenced from elsewhere. As it is a quite comprehensive article, it’s split up in separate parts. If someone (i.e., a journal editor) is interested in publishing this article, yes, I can still do that! :-)

Part I was about giving an introduction to the topic and an overview on multimedia content adaptation techniques. Part II was about the adaptation by transformation approach that utilizes scalable coding formats such as JPEG2000, MPEG-4 BSAC, and MPEG-4 SVC. This part comprises adaptation decision-taking also known as the brain of multimedia content adaptation.

Part III – Adaptation Decision-Taking

Definition: (Multimedia) adaptation decision-taking is referred to as the process of finding the optimal parameter settings for (multiple, possibly in series connected) multimedia content adaptation engines given the properties, characteristics, and capabilities of the content and the context in which it will be processed.

Problem Description

The information revolution of the last decade has resulted in a phenomenal increase in the quantity of multimedia content available to an increasing number of different users with different preferences who access it through a plethora of devices and over heterogeneous networks. End devices range from mobile phones to high definition TVs, access networks can be as diverse as UMTS (Universal Mobile Telecommunications System) and broadband networks, and the various backbone networks are different in bandwidth and Quality of Service (QoS) support. Additionally, users have different content/presentation preferences and intend to consume the content at different locations, times, and under altering circumstances, i.e., within a variety of different contexts.

In order to cope with situations indicated above, multimedia content adaptation has become a key issue which results in a lot of research and standardization efforts collectively referred to as Universal Multimedia Access (UMA). An important aspect of UMA is adaptation decision-taking (ADT) which aims at finding the optimal parameter settings for the actual multimedia content adaptation engines based on the properties, characteristics, and capabilities of the content and the context in which it will be processed. This article provides an overview of the different metadata required for adaptation decision-taking and points out technical solution approaches for the actual decision-taking.

High-level Architecture and Metadata Assets

Figure 1 depicts a high-level architecture for adaptation decision-taking including the actual content adaptation. The input of the adaptation decision-taking engine (ADTE) can be divided into content- and context-related metadata. The former provides information about the syntactic and semantic aspects (e.g., bitrate, scene description) of the multimedia content that support the decision-taking process. The latter describes the usage environment (e.g., terminal capabilities) in which the multimedia content is consumed or processed. The result of the ADTE is an adaptation decision which steers the multimedia content adaptation engine(s) to produce the adapted multimedia content fulfilling the constraints imposed by the context-related metadata. The input to the actual adaptation engine is the given multimedia content possibly accompanied with additional content-related metadata required for the adaptation itself (e.g., syntax descriptions).

 

Figure 1. High-level Architecture of Adaptation Decision-Taking and Multimedia Content Adaptation.

The focus of this article is on the ADTE. In the following sections the two types of metadata assets required for adaptation decision-taking are reviewed and, finally, technical solution approaches are highlighted.

Content-related Metadata

This type of metadata comprises descriptive information about the characteristics of the content which can be divided into four categories:

• Semantic metadata provides means for annotating multimedia content with textual information enabling various applications such as search and retrieval. This kind of metadata covers a broad range of annotation possibilities, among them are the name of the content, authors, actors, scene descriptions, etc.
• Media characteristics describe the syntactical information pertaining to multimedia bitstreams in terms of the physical format and its characteristics. This may include the storage and coding format as well as bit-rate, frame-rate, width, height, and other related parameters.
• The Digital Rights Management (DRM) information for adaptation decision-taking specify which kind of adaptation operations (e.g., scaling, format conversion, etc.) are allowed and under which constraints (e.g., bit-rate shall be greater than 512kbps).
• Finally, Adaptation Quality of Service (QoS) describes the relationship between usage environment constraints, feasible adaptation operations satisfying these constraints, and associated utilities (i.e., qualities).

Context-related Metadata

Similar to the content-related metadata, the context-related metadata can be also divided into four categories:

• End user-related metadata: The first category of metadata is pertained to metadata describing the characteristics of end users in terms of preferences, disabilities, and location-based information.
• Terminal-related metadata: The second category provides context information regarding the capabilities of the terminal which are used by the end users for consuming multimedia content. This information includes – among others – information about the codecs installed, display properties, and audio capabilities.
• Network-related metadata: The third category of metadata comprises the information concerning the access and core networks in terms of its characteristics and conditions. Such information may include bandwidth, delay, and jitter.
• Adaptation-related metadata: Finally, the fourth category of metadata describes the actual adaptation engines in terms of adaptation operations they are capable to perform. For example, an adaptation engine may be able to perform temporal scaling whereas another one provides means for spatial scaling or even complex transcoding operations between different coding formats.

Solution Approaches for Adaptation Decision-Taking

In the literature the following approaches towards adaptation decision-taking are known:

• Knowledge-based ADT [1]: adopts an Artificial Intelligence-based planning approach to find an appropriate sequence of adaptation steps from an initial state (i.e., described by content-related metadata) towards a goal state (i.e., described by context-related metadata).
• Optimization-based ADT [2]: models the problem of finding adaptation decisions as a mathematical optimization problem by describing functional dependencies between content- and context-related metadata. Furthermore, limitation constraints as well as an objective function is specified.
• Utility-based ADT [3]: can be seen as an extension of the previous two approaches which explicitly takes the users’ specific utility aspects into account.

This is the end of Part III and I will continue in Part IV with interoperability issues, i.e., standards supporting UMA. Thus, stay tuned!
 

References

[1]    D. Jannach, K. Leopold, C. Timmerer, and H. Hellwagner, "A Knowledge-based Framework for Multimedia Adaptation", Applied Intelligence, vol. 24, no. 2, pp. 109-125, April 2006.
[2]    I. Kofler, C. Timmerer, H. Hellwagner, A. Hutter, and F. Sanahuja, "Efficient MPEG-21-based Adaptation Decision-Taking for Scalable Multimedia Content", Proceedings of the 14th SPIE Annual Electronic Imaging Conference – Multimedia Computing and Networking (MMCN 2007), San Jose, CA, USA, January/February 2007.
[3]    M. Prangl, T. Szkaliczki, and H. Hellwagner, "A Framework for Utility-based Multimedia Adaptation", IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 6, pp. 719-728, June 2007.