I've co-edited the December 2010 theme of
IEEE Computing Now on
Multimedia Metadata and Semantic Management.
Guest Editors' Introduction • Harald Kosch and Christian Timmerer • December 2009
Multimedia semantics is more than developing ontologies to describe the nature of multimedia content. It’s the key research area for interoperable, intelligent access to and management of multimedia materials.
There are many metadata standards. More than 10 organizations vie for leadership in content description, including the Dublin Core Metadata Initiative, ISO/IEC’s MPEG working group, and the World Wide Web Consortium (W3C). For a complete list, see the “Semantic Standards” sidebar.
Recent studies show that this diversity is a major hindrance to a common multimedia semantic understanding. So, the first challenge to address in this area is the heterogeneity in metadata description and query languages. We must build better bridges across semantic gaps. We also need to cleverly aggregate and concisely present results for users while providing security and related access-control techniques appropriate to multimedia content. Other challenges include synchronizing metadata information to media and vice versa and managing this relationship throughout the metadata life cycle.
Effective multimedia management must span the metadata life cycle—from its creation through processing, storage, distribution, and deployment—and work whether the metadata is tightly connected with or independent of the media it describes.
Finally, we need better integration of situational context. This includes not only domain knowledge, but also legal and cultural issues, metadata and semantic quality, compression and encryption techniques.
Combining the Semantic Web with multimedia semantics offers interesting research opportunities for social-information management, such as collaborative multimedia tagging, semantics-aware social-media engineering, and multimedia mash-ups. These opportunities were well represented at the 2009 International Conference on Semantic and Digital Media Technologies (SAMT 09, www.samt2009.org). The Virtual Campfire exemplifies emerging systems for integrating social multimedia. This project, led by Ralf Klamma at the RWTH Aachen (www.dbis.rwth-aachen.de/lehrstruhl/projects/virtualCampfire), establishes an advanced framework to create, search, and share multimedia artifacts with context awareness across communities.
Selected Articles on Multimedia Semantics
This month’s theme includes the following featured articles:
In “Managing and Querying Efficiently Distributed Semantic Multimedia Metadata Collections” (IEEE MultiMedia, Oct.–Dec. 2009, pp. 12–20, special issue on Multimedia Metadata and Semantic Management), Sébastien Laborie, Ana-Maria Manzat, and Florence Sèdes propose an original model of a centralized metadata resume. Their resume is a concise version of the whole metadata, and it can link to some desired multimedia content on remote servers and databases. The authors also propose an automatic construction process for the metadata resume. They demonstrate the framework with current Semantic Web technologies for representing and querying semantic metadata. Their experimental results show the benefits of their approach.
In “Semantic MPEG Query Format Validation and Processing,” also from IEEE MultiMedia’s special issue (Oct.–Dec. 2009, pp. 22–33) Mario Doeller, Ruben Tous, Matthias Gruhne, Miran Choi, Tae-Beom Lim, Jaime Delgado, and Armelle Yakou describe the semantic validation of the MPEG Query Format (MPQF) and the implementation of an MPQF engine on top of an Oracle database management system. MPQF enables interoperable querying among heterogeneous databases that use different metadata standards for describing multimedia content. This article introduces methods for evaluating MPQF semantic-validation rules not expressed by syntactic means within the XML Schema used by the databases. The authors highlight a prototype implementation of an MPQF-capable processing engine using QueryByFreeText, QueryByXQuery, QueryByDescription, and QueryByMedia query types on a set of MPEG-7 based image annotations.
In “Using Social Networking and Collections to Enable Video Semantics Acquisition,” a third article from IEEE MultiMedia’s special issue (Oct.–Dec. 2009, pp. 52–60), Stephen Davis, Ian Burnett, and Christian Ritz consider the multimedia value chain’s first elements: media production, acquisition, and metadata gathering. The authors bring together methods from video content annotation and social networking to solve problems associated with gathering metadata that describes user interactions with and opinions about video content. Then they aggregate individual users’ interaction metadata to form semantic metadata for a given video. The authors have successfully implemented their techniques in a custom Flex application based on the popular Facebook API.
In “The Ariadne Infrastructure for Managing and Storing Metadata” (IEEE Internet Computing, July/Aug. 2009, pp. 18–25) issue of Stefaan Ternier, Katrien Verbert, Gonzalo Parra, Bram Vandeputte, Joris Klerkx, Erik Duval, Vicente Ordóñez, and Xavier Ochoa analyze the standards-based Adriane infrastructure for managing learning objects in an open, scalable architecture. The core infrastructure comprises several components such as the repository, federated search engine, finder, harvester, and metadata validation service. This infrastructure enables the integration of learning objects in multiple, distributed repository networks. Finally, the authors review several architectural patterns that they found useful in searching repositories in this area—namely, federated search, search on harvest, search adapter, and harvest adapter. It would be interesting to see this infrastructure working multimedia metadata.
In “Data-Sharing P2P Networks with Semantic Approximation Capabilities” (IEEE Internet Computing, Sept./Oct. 2009, pp. 60–70), Federica Mandreoli, Riccardo Martoglia, Simona Sassatelli, and Wilma Penzo tackle the new information-retrieval challenges posed by heterogeneous data representations within peer-to-peer systems. The authors suggest leveraging the presence of semantic approximations between peers’ schemas to improve query routing. Their approach identifies the peers that best satisfy a user’s query and ranks the answers through a mechanism that promotes the most semantically relevant results. Their work applies to a scenario in which various actors in a multimedia chain-of-value network (such as network and telecom operators and service providers) must actively collaborate.
In “3D Media and the Semantic Web” (IEEE Intelligent Systems, Mar./Apr. 2009, pp. 90–96), Michela Spagnuolo, and Bianca Falcidieno introduce ways to integrate 3D media with Semantic Web technologies. Tools for coding, extracting, sharing, and retrieving the semantic content of 3D media are still far from satisfactory. The authors describe a means for embedding 3D into the Semantic Web, documenting and annotating 3D media for sharing, understanding its meaning, and retrieving it on the basis of content.
Related Resources
Numerous other articles from a wide range of journals and conferences deal with topics related to Multimedia Semantics; see our accompanying list of recommendations.
We’d also like to know what you think about Multimedia Semantics, so take this month’s poll and voice your opinion.
Harald Kosch is a full professor at the Faculty of Informatics and Mathematics, University of Passau, Germany. His research interests include multimedia metadata, multimedia databases, middleware, and Internet applications. Kosch has a PhD in computer science from Ecole Normale Supérieure de Lyon, France. Contact him at Harald.Kosch@uni-passau.de.
Christian Timmerer is an assistant professor in the Department of Information Technology, Multimedia Communication Group, Klagenfurt University, Austria. His research interests include the transport of multimedia content, multimedia adaptation in constrained and streaming environments, distributed multimedia adaptation, and Quality of Service/Quality of Experience. Timmerer has a PhD in applied informatics from Klagenfurt University. Contact him at christian.timmerer@itec.uni-klu.ac.at. Publications and MPEG contributions can be found under http://research.timmerer.com, follow him on http://www.twitter.com/timse7, and subscribe to his blog http://blog.timmerer.com.