Lets take a look at the progress in MICO and especially the enabling technologies that compose the MICO platform for cross-media analysis, querying and recommendations. For the purposes of consistency we will call the MICO platform components “MICO Enabling Technology Modules”. The modules consist of a basic set of extractors dynamically orchestrated according to the content as required for the use case evaluation phase of MICO.
This update is based on the official project report (deliverable) that is publicly available for download at this link or under the section publications.
The document summarises the following results and progress in the project:
- First version of the prototype for orchestrating extraction components.
- Prototype implementation of cross-media publishing in Apache Marmotta, including analysis results and media metadata representation.
- Prototypical implementation of the cross-media query language in Apache Marmotta.
- Prototype implementation of the cross-media recommendations.
Service Orchestration – for managing cross-media analysis: A central part of the MICO project is the service orchestration for extractors and components. The service orchestration component is expected to compute each input content item towards an appropriate individual execution plan that is then followed by the available analysis components. This high-level goal is approached by following an iterative approach, where this first iteration delivers a broker with the idea of learning by experimenting, and then proposing changes and evolution towards providing a full orchestration solution by the next iteration.
Multimedia Metadata Model: Multimedia analysis components typically operate in isolation as standalone applications and therefore do not consider the context of other analyses of the same media resource. We document here a multimedia metadata model in conjunction with a metadata API. The metadata model and API can be adapted for third party extractors.
Cross-media querying approach with SPARQL-MM: With the introduction of the complex annotation model both annotators and humans can put media and concepts in context. The model represents information using RDF, which makes SPARQL a good candidate for an adequate retrieval mechanism / query language. Nevertheless some functionalities that are necessary for proper media asset and fragment retrieval are missing. Here we document Multimedia Extension for SPARQL named SPARQL-MM. The extension includes mainly relation and aggregation functions for media fragments but is under continuous development so this description is just a snapshot of the current version.
Cross-media recommendations: Recommender systems have changed the way people find information. Based on behaviour patterns and content analysis, items can be presented to the users that might be completely new to them, but match their expectations. Within MICO, the goal and opportunity is to use various metadata types, and apply recommendation algorithms to further enrich and extend metadata, creating richer models which can be used for various recommendation purposes. The underlying theme of both showcases in MICO, is the need for cross-media recommendation: Automatic and manual annotations, contextual information and user interaction data related to various media types can be used as input to the algorithms, resulting in recommendations for all relevant media types as output, thereby crossing media borders. For instance, a user preference for images with lions (which may have been identified by automatic image analysis or manual annotation, and usage information) can be used to recommend related images, documents, posts, videos or video fragments. Consequently, one of the key challenges for this domain is to define which and how to use information sources to calculate similarity as needed for a specific use case.
MICO Platform for Cross-media analysis, querying and recommendations: This document also provides instructions on how to setup and run the MICO Platform. The MICO Platform Server runs on a Linux server providing the following MICO relevant services: Apache Marmotta with contextual extensions, Broker, RabbitMQ, Hadoop HDFS server for binary content storage. For more information on how to install and run the MICO platform please see the section MICO Release, and follow the discussion in the News thread.