What is cross media recommendation?

Within MICO, we define Cross Media recommendation by looking at the inputs and outputs of a recommendation engine. What is the type of the data analysed? What is the type of the items recommended?

A recommendation is cross media if

• the type of input and output (e.g., video, text, images, audio) differ
• various types of input are combined to calculate a recommendation.

Note that cross media does not imply cross-modal analysis (an analysis of audio and video within the same item) but will surely profit from it in certain use cases.

For the MICO project, the goal is clear: combine different extractors that were developed within the project and calculate meaningful recommendations out of it. In the course of the project, we developed together with our showcase partners two scenarios which are currently in development. The aim of this blog post is to briefly define the scenarios and to have a place to link back to when explaining the implementation without having to introduce the motivation again and again.

Btw: In MICO there is more than cross media recommendations. We also work on recommendation using collaborative filtering techniques using Prediction.io – this will be covered in a later blog post.

The other two recommendation goals are focused on collaborative filtering using prediction.io and will be discussed in a next post.

Greenpeace Magazine Editor Support

Editing articles for online platforms is different from writing articles for traditional press. The reader expects that the possibilities offered by the new medium are utilized.

However, writing an article is one thing, enriching it with different media types another. In a first step, the writer will add content it knows, but it would certainly profit from a recommendation that proposes fitting content, and the task of the editor is to choose from the offered content.

As the editor has full control what content is linked (in contrast to some automatic systems), there will be basically no false positives. On the other hand, if the underlying recommendation system is good enough, the editor saves a lot of time searching for content and can spend more time doing original research.

In MICO, we do this for the italian Greenpeace Magazine which is run on a WordPress/Wordlift system. We will analyse the pool of Greenpeace-produced videos using our Speech-To-Text and NER components. This will be matched to a analysis of the article that is currently written to suggest videos with fitting content. If the video contains textual metadata, those will be analyzed. But even if not: the content analysis allows us to search archives without human interaction.

Zooniverse Discussion Analysis

For the Zooniverse showcase, things are less straightforward. As the recommendation of subjects to the users might has counterintuitive and maybe negative impact on the crowd sourcing performance, we had to search for an other opportunity to employ cross media recommendation.

In Zooniverse Projects, users can discuss publicly about subjects to help themselves classifying. For Snapshot Serengeti it is common that some users post snippets like #wildebeest, but also complex discussions occur. In MICO we will analyse the text, using our NER and competence classifications and combine this with the information we have on the subject (human annotations and extractor data). As a result we can automatically recommend training material or reference images that will help the user to make more confident annotations.

Both the Greenpeace and the Zooniverse case rely on the same basic architecture which will be presented at the 4th workshop on Linked Media at this year’s ESWC:

Thomas Köllmer, Emanuel Berndl et al.: A Workflow for Cross Media Recommendations based on Linked Data Analysis.