Publications

Volume 1: State of the Art in Cross-Media Analysis, Metadata Publishing, Querying and Recommendations

sota_sept2015_coverResponsible editors:
Patrick Aichroth, Johanna Björklund,
Florian Stegmaier, Thomas Kurz, Grant Miller

Issue: August, 2015 |
ISBN 978-3-902448-43-9

Click here for download

This report represents the combined state of the art for work packages WP2, WP3, WP4, and WP5 of the MICO project. Its main objective is to provide the persons working on models and components in later phases of the project with a single entry point to lookup relevant conceptual background as well as concrete technologies as needed while implementing the generic MICO framework and particularly the two use cases. For this reason, the areas covered by this document have been carefully selected in accordance with the requirements gathering in work packages WP7 and WP8, and the system architecture in WP6. Any deliverables that are referenced within this report are free accessible on the web (https://www.mico-project.eu/publications/).

To summarize, the Zooniverse use case in WP7 will be mostly concerned with “object detec- tion” and specifically “animal detection” in the Snapshot Serengeti1 (SC-16), Plankton Portal2 (SC-10) and Galaxy Zoo3 (SC-14) projects projects, while the IO10 use case in WP8 will address quality assessment and annotation of commercial news video content (SC-02) and music content (SC-01). The reader is referred to MICO deliverables D7.1.1 and D8.1.1 for more details on the use cases. As the detailed requirements gathering is not fully completed, this report opens paths to different approaches in most areas, and is therefore very extensive. However, throughout the report there are many references pointing to the relevance to one or both of these use cases.

Since the report is a joint work of several work packages, we structured it mainly according to the tasks defined in the MICO Description of Work:

  • Section 2 (Cross-Media Analysis) gives a thorough introduction into different approaches to analysing and extracting knowledge from media content and is the main outcome of work package WP2. In particular, it describes how analysis results are typically represented for further process- ing, it summarizes typical approaches to text analysis and extraction, and gives an overview over audio-visual analysis. Since the area is very broad and highly dependent on the actual domain, specific focus has been given to “animal detection” and A/V quality assessment, as these are the main topics of the two use cases.
  • Section 3 (Metadata Publishing) is specifically concerned with the question how media content can be annotated, how analysis services can be described and how to represent trust and prove- nance in the context of analysis metadata and annotations. Since the MICO project is mostly concerned with Web content, the state of the art in this section focusses mainly on Semantic Web technologies to achieve these purposes. This section is the main outcome of work package WP3.
  • Section 4 (Multimedia Querying) summarizes approaches for multimedia querying. It first gives an overview over different multimedia query languages that have been developed in the context of relational and semi-structured databases. In accordance with Section 3, it then describes the Semantic Web query language SPARQL and its extension mechanism, serving as the foundation for the MICO multimedia query language. This section is the main outcome of work package WP4.
  • Section 5 (Multimedia Recommendations) investigates how to use the results from cross-media analysis, metadata publishing and multimedia querying for recommendations. It gives a summary

    of different approaches for recommender systems and specifically highlights those approaches that might be useful for the two use cases. This section is the main outcome of work package WP5.

    • Section 6 (Implementations) finally gives a comprehensive list of concrete implementations that are available for many of the conceptual approaches described in the four main sections. It can serve as a reference for quickly looking up candidate technologies when needed.

    Note that while these Sections follow mostly the respective work packages, the different areas have influenced each other significantly through the collaborative editing of the report. As most authors have also participated in the requirements analysis in work packages WP7 and WP8, the technologies and approaches described here should be of high relevance to the upcoming work in the MICO project.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Volume 4: Specifications and Models for Cross-Media Extraction, Metadata Publishing, Querying and Recommendations- Final Version

mico_publication_4Issue: December, 2015 |
ISBN: 978-3-902448-46-0

Click here for download

This report provides the final specification draft for work packages 2 to 5 in MICO. Based on the initial specification from [Aic+15c], and considering the first software testing and evaluation results from our showcase parters InsideOut10 and Zooniverse, it covers all updates and newly introduced extractors and pipelines, and specifies the next version of the MICO broker, which will be implemented in Year 3. Moreover, it describes version 2 of the annotation model, and finally provides an update to the querying and recommendation capabilities for the MICO platform.

Section 2 introduces the textual, visual and audio extractor enhancements and updates, and describes current pipelines and pipelines planned for Year 3. Moreover, it outlines the technical requirements for extractor orchestration, relevant broker updates until now, and provides the broker v3 design to be implemented in Year 3.

Section 3 describes the updated ontology for multimedia metadata (MICO MetadataModel1). This includes a summary of the model basics, lessons learned from the first two projects years, and how they lead to version 2 of the metadata model, which includes OWL specifications and descriptive docu- mentation for improved usability of the model. In addition, a dedicated ontology has been designed to incorporate MICO showcase-specific needs. Finally, a description of Anno4j2 is provided, which is a software designed to simplify RDF and SPARQL usage via MICO annotations.

Section 4 delivers an extended specification of SPARQL-MM, a multimedia query language. It in- cludes additional fragment accessor functions, an extension of the Media Fragment URI standard, a description of GeoSPARQL features and a introduction to SPARQL Inferencing Notations. Further- more, it includes an outlook to features to be implemented in Year 3.

Section 5 describes the recommendation framework for MICO, which aims at using selected results of cross-media analysis with user data, to provide cross-media recommendation. After providing the updated recommendation user stories and requirements, it describes the framework architecture, with a focus on the different needs of the relevant showcases Shoof, Greenpeace Magazine and Snapshot Serengeti. Moreover, the section includes the specification of interfaces and component interactions and a comparison with the project deliverable D5.2.1, and a description of available data sources.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Combined Use Cases: First Prototypes

Combined Use Cases: First Prototypes
May 2015

Combined-Deliverable7.2.1-8.2.1UseCasesFirstPrototype1Screen Shot 2016-04-15 at 19.58.05This document outlines the Use Case Prototype setups by the use case partners (Zooniverse and Insideout10) for the evaluation of the MICO platform.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download

Use Cases: Validation Report First Evaluation 2015

Use Cases: Validation Report First Evaluation 2015
October 2015

Screen Shot 2016-04-15 at 19.46.19This document outlines the Test Plan (First Evaluation) describing the setup of the first evaluation round in the two use cases (WP7 and WP8) and the functionalities that are evaluated. It is currently restricted to the testing conducted during the development of the MICO platform. The aim is to compare each MICO Technology Enabler (TE) prior to beginning end­-to-­end testing of the system.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download

Use Cases Requirements Anaylsis

Compendium Use Case Requirements AnalysisScreen Shot 2016-04-15 at 19.34.41

This document summarizes the high­ level requirements for the MICO project, which serve as a basis for all further R&D steps. The requirements result from an exchange between all partners within the consortium, aligning the user perspective and technical possibilities.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download

MICO System Architecture and Development Guidelines

MICO System Architecture and Development GuidelinesScreen Shot 2016-04-15 at 19.31.19

30 April 2014

Abstract: This deliverable describes an initial architecture for the MICO platform based on high-level requirements known already at the beginning of the project. It focusses on the core architecture, i.e. infrastructure, communication, persistence, orchestration, and technological decisions, as well as on development infrastructure and best practices, and is supposed to give guidance on how to start development of the platform. Requirements currently gathered in WP7/WP8 and specifications under development in WP2-WP5 will be taken into account in a revised version of this document.

Keywords: architecture, best practices, infrastructure

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download

Volume 2: Specifications and Models for Cross-Media Extraction, Metadata Publishing, Querying and Recommendations

mico_publication_2Issue: October, 2015 |
ISBN: 978-3-902448-44-6

Click here for download

This report provides an initial specification draft for Work Packages (WP) 2-5 in the Media in Context (MICO) project. At its core, it provides a standard ontology for publishing metadata based on several existing ontologies, adapted and extended to the MICO requirements, now available at https://www.mico-project.eu/ns/platform/1.0/schema#. This is complemented by an annota- tion design based on the Open Annotation Data Model, including examples.

A first set of prioritized textual, visual, and audio extractors is presented. These are aimed at the first two MICO showcases from InsideOut10 and Zooniverse. For each of them, design choices and evalu- ation criteria are outlined, which serve as a basis for future testing. The input and output is described and the resulting annotation design is derived from that, which provides the basis for the integration of extraction results. In addition, potential interactions with other extractors have been identified. Even within this initial set of extractors it has become obvious that there is significant potential for improving performance regarding quality and completeness through a combination of approaches.

Regarding querying, an extended version of the query language SPARQL is presented. Besides basic SPARQL functionality like full read-write support for structured data and novel path patterns, it supports fulltext search for several languages as well as spatio-temporal query functions. To improve the efficiency and to build a proper basis for further extensions, MICO provides a SPARQL-to-SQL translation, which is integrated in the open source software Apache Marmotta.

Several cross-media recommendations approaches are presented (content-based, collaborative filter- ing, ontology-based) and their technical specifications are introduced. The proposed recommendation models extend classical recommender system approaches with the aim of taking advantage of the se- mantic features provided by the extractors and the multimedia extension of SPARQL. The benefits are a higher accuracy of the content suggestions and the possibility to group users with similar traits.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Volume 3: MICO Enabling Technology ModulesA design system that includes a massive array of elements.

mico_publication_3Issue: November, 2015 |
ISBN: 978-3-902448-45-3

Click here for download

This technical report outlines the initial implementation of the MICO Enabling Technology Modules. The modules consist of a basic set of extractors dynamically orchestrated according to the content as required for the use case evaluation. The report will focus primarily on adapting existing extractors. It will include both open source as well as proprietary extractors. The document summarises the follow- ing results and progress in the project relative to the initial specifications (see Volume 2 of this report series):

  • First version of the prototype for orchestrating extraction components.
  • Prototype implementation of cross-media publishing in Apache Marmotta, including analysis re-sults and media metadata representation.
  • Prototypical implementation of the cross-media query language in Apache Marmotta.
  • Prototype implementation of the cross-media recommendations.

A central part of the MICO project is the service orchestration for extractors and components. The service orchestration component is expected to compute each input content item towards an appropriate individual execution plan that is then followed by the available analysis components. That high-level goal is approached in MICO following an iterative approach, where this first iteration delivers a broker with the idea of learning by experimenting, and then proposing changes and evolution towards providing a full orchestration solution by the next iteration.

Multimedia analysis components typically operate in isolation as standalone applications and there- fore do not consider the context of other analyses of the same media resource. We document here a multimedia metadata model in conjunction with a metadata API. The model will be introduced in Section 3.1, the decision taking and a description of the API in Section 3.2. Section 3.3 will cover the extensibility of the model and the API, allowing the model to be adapted for third parties and their extractors.

With the introduction of the complex annotation model both annotators and humans can put media and concepts in context. The model represents information using RDF, which makes SPARQL a good candidate for an adequate retrieval mechanism and query language. Nevertheless some functionalities that are necessary for proper media asset and fragment retrieval are missing, like described in [MICO2]. Here we document Multimedia Extension for SPARQL named SPARQL-MM. The extension includes mainly relation and aggregation functions for media fragments but is under continuous development so this description is just a snapshot of the current version.

Recommender systems have changed the way people find information. Based on behaviour patterns and content analysis, items can be presented to the users that might be completely new to them, but match their expectations. Within MICO, the goal and opportunity is to use various metadata types, and apply recommendation algorithms to further enrich and extend metadata, creating richer models which can be used for various recommendation purposes.

The underlying theme of both showcases in MICO, is the need for cross-media recommendation: Automatic and manual annotations, contextual information and user interaction data related to various media types can be used as input to the algorithms, resulting in recommendations for all relevant me- dia types as output, thereby crossing media borders. For instance, a user preference for images with lions (which may have been identified by automatic image analysis or manual annotation, and usage

information) can be used to recommend related images, documents, posts, videos or video fragments. Consequently, one of the key challenges for this domain is to define which and how to use information sources to calculate similarity as needed for a specific use case. This document also provides instructions on how to setup and run the MICO Platform.

The MICO Platform Server runs on a Linux server providing the following MICO relevant services: Apache Marmotta with contextual extensions, Broker, RabbitMQ, Hadoop HDFS server for bi- nary content storage. For development and testing purpose, we provide a ready-to-use virtual image (see Section 6.1). If you would like to install the MICO Platform on your own machine, take a look at Section 6.2.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Volume 5: MICO Enabling Technology Modules - Final Version

mico_cover-report5Issue: November, 2016 |
ISBN: 978-3-902448-47-7

Click here for download

This document provides the final description of the MICO technology enablers from WP2, WP3, WP4 and WP5, including several system domain updates (model / persistence, broker, querying, and recommendation framework) since Year 2.

Section 2 provides the relevant updates regarding MICO extractors and their implementation status with respect to the specifications of (Mico Technical Report V4) related to linguistic analysis, including diarization, ASR, text and competence classification, language detection, and NER. It also includes a description of two new extractors for animal vs. blank image distinction in Year 3. Finally, Section 2 describes how the MICO extractor API evolved since the MICO Technical Report Volume 3, and how to implement a MICO extractor.

Section 3 describes the MICO broker v3 for extractor orchestration, which represents a major up- date over v1 provided with report V3, and also over v2 that was provided earlier in Year 3. The section outlines requirements and design principles, describes the broker data model, provides a component overview, and presents the registration service and its interplay with extractors. It includes an expla- nation of how semi-automatic creation and workflow execution has been implemented. Section 3 also provides a description of workflow management tools that were not planned for Year 3, and are not part of and not needed to use the platform, but nonetheless improve usability.

Section 4 provides an updated description of Anno4j, the Object-to-RDF ORM mapping for MICO that allows the creation of MICO or generic W3C Web Annotations with plain Java objects. Anno4j was originally introduced in report V3, and has been significantly extended since then.

Section 5 updates the description of the query language extension SPARQL-MM to version 2.0. It then continues to describe how LDPath, a simple path-based query language, similar to XPath, can be used for simplified querying and data retrieval. LDPath is here provided as an alternative to SPARQL- MM, which was used in earlier reports. The conclusion of Section 5 presents a solution approach to the problem of Semantic Media Similarity.

Section 6 describes the MICO recommendation framework, including status and usage instructions for the related components, and a description of the recommendation API and respective demo code.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

MICO Platform Initial Version

MICO Platform Initial VersionScreen Shot 2016-04-15 at 19.38.31
31 October 2014

Abstract: This deliverable contains the initial implementation the MICO platform and integration of the enabling technology components from WP2-5. The current version is the first of three iterations. It provides an implementation of the initial architecture proposed in D6.1.1, as well as the early evolution of some aspects within the project.

 

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download

Use Cases: Test Plan First Evaluation 2015

Use Cases: Test Plan First Evaluation 2015Screen Shot 2016-04-15 at 19.44.48
May 2015

This document outlines the Test Plan (First Evaluation) describing the setup of the first evaluation round in the two use cases (WP7 and WP8) and the functionalities that are evaluated. It is currently restricted to the testing conducted during the development of the MICO platform. The aim is to compare each MICO Technology Enabler (TE) prior to beginning end­-to­end testing of the system.

MICO (Media in Context) is a research project partially funded by the European Commission 7th Framework Programme (grant agreement no: 610480).

Click here for download