Vol-3194/paper44

Paper

Paper
edit
description
id	Vol-3194/paper44
wikidataid	Q117344993→Q117344993
title	Towards Extreme Multi-Label Classification of Multimedia Content
pdfUrl	https://ceur-ws.org/Vol-3194/paper44.pdf
dblpUrl	https://dblp.org/rec/conf/sebd/MiniciPG022
volume	Vol-3194→Vol-3194
session	→

Towards Extreme Multi-Label Classification of Multimedia Content

Towards Extreme Multi-Label Classification of
Multimedia Content
(Discussion Paper)

Marco Minici1,2 , Francesco Sergio Pisani2 , Massimo Guarascio2 and Giuseppe Manco2
1
    Università degli Studi di Pisa
2
    ICAR-CNR, Rende (CS), Italy


                                         Abstract
                                         Providing rich and accurate metadata for indexing media content represents a major issue for enterprises
                                         offering streaming entertainment services. Metadata information are usually exploited to boost the
                                         search capabilities for relevant contents and as such it can be used by recommendation algorithms for
                                         yielding recommendation lists matching user interests. In this context, we investigate the problem
                                         of associating suitable labels (or tag) to multimedia contents, that can accurately describe the topics
                                         associated with such contents. This task is usually performed by domain experts in a fully manual fashion
                                         that makes the overall process time-consuming and susceptible to errors. In this work we propose a
                                         Deep Learning based framework for semi-automatic, multi-label and semi-supervised classification. By
                                         integrating different data types (e.g., text, images, etc.) the approach allows for tagging media contents
                                         with specific labels. A preliminary experimentation conducted on a real dataset demonstrates the quality
                                         of the approach in terms of predictive accuracy.

                                         Keywords
                                         Extreme Multi-Label Classification, Data Integration, Natural Language Processing, Semi-supervised
                                         Learning




1. Introduction
Nowadays, entertainment industry represents one of the most profitable and widespread busi-
ness sector, with a constant growth in terms of number of users. With estimated revenues
amounting to about 2 trillion of dollars worldwide, providing effective research services is
a crucial task for the companies operating in multimedia content delivery. In particular, the
rise of streaming services and on-demand contents fostered the interest for AI-based solutions
capable to facilitate the research and identification of contents matching the user interests. Just
as an example, Recommender Systems (RS) are technologies widely adopted by big players
(e.g., Netflix, Disney+, Amazon, etc.) to suggest items of their catalogues able to arouse users’
interest.

SEBD 2022: The 30th Italian Symposium on Advanced Database Systems, June 19-22, 2022, Tirrenia (PI), Italy
$ marco.minici@icar.cnr.it (M. Minici); giuseppe.manco@icar.cnr.it (F. S. Pisani); massimo.guarascio@icar.cnr.it
(M. Guarascio); giuseppe.manco@icar.cnr.it (G. Manco)
€ https://mminici.github.io (M. Minici)
� 0000-0002-9641-8916 (M. Minici); 0000-0003-2922-0835 (F. S. Pisani); 0000-0001-7711-9833 (M. Guarascio);
0000-0001-9672-3833 (G. Manco)
                                       © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
    CEUR
    Workshop
    Proceedings
                  http://ceur-ws.org
                  ISSN 1613-0073
                                       CEUR Workshop Proceedings (CEUR-WS.org)
�   Besides them, the technologies that allow for enriching content metadata with informative
labels (or tags) act a key role as they can exploited to improve the RS performances and
simultaneously enable a more effective research by means of the traditional research engines.
Basically, these labels are used to group contents exhibiting common features and provide
aggregated views for the users. However, the labelling task is a time-consuming and prone
to the error process since it is manually performed by domain experts. Indeed, the lack of
a common shared taxonomy can lead to yield repeated labels describing the same concept.
Moreover, the assignment of a label to a content is subjective and depends on the skill and
perception of the expert.
   In this scenario, Artificial Intelligence (AI) techniques represent a valuable tool to automate
such a process by limiting the human factor and, as a consequence, reducing the classification
error. Anyway, effectively addressing this problem requires the development of specific ap-
proaches able to cope with different hard issues, i.e., unbalancing of the classes, lack of labelled
data, capability of the models to process different types of data (e.g., text, images, etc.) and
providing multi-class predictions on an high number of labels.
   In particular, Deep Learning (DL) paradigm [1] is considered a state-of-the-art solution for
effectively address these issues. DL-based models can be exploited to extract accurate multi-
label classification models by combining raw low-level data, gathered from a wide variety
of sources (e.g., wikidata, IMDB, etc.). These models learn in a hierarchical fashion: several
layers of non-linear processing units are stacked in a single network and each subsequent
layer of the architecture can extract features with a higher level of abstraction compared to the
previous one. Therefore, Deep Learning-based approaches allow to extract data abstractions
and representations at different levels, they also represent a good choice for analyzing raw data
provided in different formats and by different types of source.
   In this work we propose to combine different types of data from different publicly available
data sources for classifying media contents and enrich them with informative labels. In Figure 1,
we sketched the overall learning process. After an Information Retrieval phase in which data are
gathered and wrapped in a single view, these raw data are provided as input to Machine Learning
block. Our solution adopts a hierarchical Deep Learning based approach: on top, an ensemble
of pre-trained models (Embedder) are fine-tuned and used to map the input (text and/or images)
in a low-dimensional space. Here, the main idea is that contents with similar labels generate
similar vector representations (embeddings). Then, a clustering algorithm (Clusterer) is used to
group similar contents and yield sub-samples of the original dataset. Finally, each sub-sample
is exploited to learn a local model focused on a limited set of labels that allows for yielding
more accurate predictions for specific cases. Although our approach is totally general and
capable to handle different type of data by adding specific models to the Embedder, the current
implementation focuses on analyzing text data. An experimental evaluation conducted on a real
dataset containing movie plots demonstrates the quality of our approach in providing accurate
predictions in this challenging scenario.
   The rest of this paper is organized as follows: in Section 2 we provide an overview of the main
approaches proposed in literature to tackle the automatic content tagging problem. In Section 3,
we describe the framework used to address the problem and the deep learning architecture used
to learn the multi-label classification model; while in Section 4 we discuss the experimental
results. Section 5 concludes the work and introduces some new research lines.
�Figure 1: Overview of the Learning Process.


2. Related Work
The problem of classifying movies is not new in the literature and can be considered a general
classification task on heterogeneous (video, images, audio, text) data. Wu et al. [2] process user
reviews to extract relevant tags for movies. Afterward, they propagate these tags to less popular
products according to the movie similarity based on multiple attributes (e.g.: title, summary).
Hence, this work draws from the collaborative recommendation paradigm, while our proposal
exploits deep metric learning and content-based techniques to solve the tag sparsity problem.
Arevalo et al. [3] employ a neural architecture - inspired by recurrent units such as LSTM -
named Gated Multimodal Unit (GMU) to effectively combine features coming from the poster
image and the plot synopsis. They focus on solving the multi-modal fusion problem rather than
the movie tagging itself. Indeed, their dataset contains fewer tags than ours.
   The work that most resembles our approach is [4], which makes use of plot synopses to
predict tags in the realm of movies. They focus on modeling the plot text as an emotion flow -
i.e.: a series of consecutive states of emotion. Their main conclusion is that incorporating the
emotion flow increases the tag prediction quality with respect to naive approaches.
   Wehrmann and Barros [5] analyze movie trailers for performing multi-label genre classifica-
tion. They explore the extraction of the audio and image features to establish spatio-temporal
relationships between genres and the entire trailer. Similar to our approach, different learners
are combined. Standalone models are trained separately for the image and the audio input, then
they are fused using a weighted average. Anyhow, as stated by authors, the main limitation of
the work relies in the use of only nine common movie genres.
   Fish et al. [6] highlight how a single movie genre hold back a large semantic that can be
exploited to have a fine-grained description of the movie. The proposed model combines the
embeddings produced by four pre-trained multi-modal ‘experts’ processing the audio and video
of the movie. The training process is intended to improve the quality of the embeddings, i.e.
the similarity between each movie clip and one of the 20 genres of the tags.
   Table 1 summarizes the most significant approaches among those described above. Compared
to these approaches, there are some major differences with regards to the problem we aim to
tackle: first, the tagging task is relative to a high number of labels. Second, this large number of
labels exhibits a long-tail distribution, as illustrated also in Figure 4 and discussed later in the
paper. To the best of our knowledge, our solution is the first approach that can handle large
� Approach              Dataset          Number of tags   DL architecture                           Data Type      XMLC   Multi-Modal   Metric         Result
 Kar et al. [4]        MPST             71               LSTM                                      Text           y      n             Micro F1       0.37
 Arevalo et al. [3]    MM-IMDb          26               Multimodal Fusion with Pre-Trained nets   Text, Image    n      y             Micro F1       0.63
 Arevalo et al. [3]    MM-IMDb          26               Multimodal Fusion with Pre-Trained nets   Text, Image    n      y             Macro F1       0.54
 Wehrmann et al. [5]   LMTD             22               Multimodal Convolutional NN               Audio, Image   n      y             Micro AUC-PR   0.65
 Wehrmann et al. [5]   LMTD             22               Multimodal Convolutional NN               Audio, Image   n      y             Macro AUC-PR   0.74
 Fish et al. [6]       MMX-Trailer-20   20               Multimodal classifiers                    Audio, Image   n      y             F1-weighted    0.60

Table 1
Analysis of current literature on Genre/Tag classification.


amounts of labels (XMLC - eXtreme Multi-Label Classification) and process different types of
data.


3. Framework
In this section we illustrate our solution and the main components of the proposed DL-based
architecture. As highlighted in Section 1, we adopted a hierarchical approach composed of
three main components, as shown in Figure 2: (i) an Embedder, devoted to summarizing the
original input into a vector representation (embedding); (ii) a cluster module (Clusterer) that
allows for identifying media with similar contents and extracting focused sub-samples of the
original dataset; and (iii) the local models that perform the final predictions.

Embedder. As mentioned above, the current implementation of our technique works on text
data (i.e., the movie plots), therefore our Embedder takes the form of a widely adopted (pre-
trained) neural network i.e., BERT (Bidirectional Encoder Representations from Transformers)
[7]. BERT is a transformer-based neural architecture able to process natural language and
trained through an algorithm including two main steps, respectively named Word Masking and
Next sentence prediction. In the former step, a percentage of the words composing a sentence is
masked and the model is trained to predict the missing terms by considering the word context i.e.,
the terms that precede and follow the masked one. Then, the model is fine-tuned by considering
a further task that allows for understanding the relations among the sentences. Basically,
given two subsequent sentences, negative examples are created by replacing the second one
with a random sentence. As regards the architecture, BERT can be figured out as a stack of
transformer encoder layers that include multiple self attention “heads” [7]. In our framework,
we use a BERT instance pre-trained on Wikipedia pages and the final embedding is obtained by
averaging the output of the last four layers of the model. Notably, our BERT instance is further
fine-tuned by adopting a Deep Metric Learning [8] based approach: three instances of the same
architecture sharing the same weights are trained against triplets ⟨𝑎𝑛𝑐ℎ𝑜𝑟, 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒, 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒⟩.
Basically, the term anchor refers the reference input whereas positive and negative represent
other examples respectively similar and dissimilar to the anchor. The goal consists of minimizing
the distance between the anchor and the positive example while, simultaneously, the distance
between the anchor and the negative one is maximized. A customized version of the triplet loss
for multi-label tasks is exploited in the learning phase. Specifically, we adopted a semi-hard
negative mining approach that filters out negative instances which share more tags with the
anchor w.r.t. the positive ones. At prediction time, only a model is used to compute the vector
�            Movie Plots                            Embedding Module               Clustering Module           Classification Module
            In 1981, at a bus stop
            in Savannah, Georgia, a man named …




                                                                      embedding
            A gang of criminals rob a Gotham
            City mob bank; the Joker manipulates
            them into murdering each other …




                                                                                                      . . .
            Amid a galactic civil war, Rebel
            Alliance spies have stolen plans to
            the Galactic Empire's Death Star …




                            …                         Triplet Loss                   K-Means                        CB Loss

Figure 2: Our proposed 3-step architecture comprises three modules: embedding, clustering, and
classification. Each component can be modified to address different goals - e.g.: different data modalities,
training strategies, or models.


representation of the input data. The main benefit of this approach relies on the possibility of
combinatorically increasing the input size and handle the lack of labelled examples.

Clusterer. A clustering algorithm is involved in the framework to group similar movies, thus
allowing the deployment of local classification models. Each cluster includes movies that share
a minimal number of tags with respect to the whole label space. Hence, this phase further
alleviates the extreme-classification problem. New instances, to be classified, are assigned to
the closest cluster on the basis of a suitable distance metric (in our case, euclidean distance is
adopted). As shown in Figure 2, we adopted the K-Means algorithm as our clusterer.

Local Models. In our framework, local models take the form of neural networks too. Specifi-
cally, we exploited the DNN-based architecture, shown in Figure 3, to provide more accurate
predictions also for minority classes. The base building block of our model includes three types
of layers: (i) a fully-connected dense layer equipped with Rectified Linear Unit (ReLU) activation
function [9], for each node composing the layer, (ii) a batch-normalization layer for improving
stability and performances of the current dense layer [10], (iii) and a dropout layer for reducing
the overfitting problem [11]. Several instances of this base component can be stacked in a
single model, in particular, in our experimentation we tested a solution with three instances.
The output layer of the architectures is equipped with a sigmoid activation function[12] and a
variable number of neurons depending on the number of labels falling on the cluster associated
with the local model. Basically, the output layer provides a class probability for each label. To
address the Class Imbalance Problem, each local model is trained by using the Class-Balanced
(CB) loss proposed in [13]. Here, the main idea consists of weighting loss inversely with the
effective number of samples per class.


4. Experimental Results
In order to assess the quality of our approach in labelling movies with related tags, we conducted
a preliminary experimentation on a real dataset extracted by fusing data from different data
sources. In particular, first we illustrate the dataset and the challenges to address for providing
accurate predictions in this scenario; then we describe the adopted evaluation protocol and
�                                                                                                Fully Connected           Batch
                                                                                                      Layer            Normalization      Dropout
                                                                                                             x1 – μ1
                                                                                           x1                  σ1
        x1
                                                                                                             x2 – μ2
                                                                                           x2                  σ2
        x2




                                                                                                                                                      DNN Blocki+1
                                                                                      ~
              Input Layer




                                DNN Block1




                                                                       Output layer
                                                          DNN Blockm
                                                                                      y                      x3 – μ3




                                             . . .
                                                                                           x3                  σ3
        x3
                                                                                                      .
                                                                                                      .
                                                                                                             xN – μN
                                                                                                      .
                                                                                           xN                  σN

        xN
                                                                                                                                         DNN Blocki


                            (a) Overview of the model.                                    (b) Layers composing a base DNN component
Figure 3: Local Model architecture: overview (a) and details of the internal structure for each base
component (b).

             Dataset               Number            of      Number      of               Avg. number of                               Avg. number of
                                   Movies                    Unique Tags                  Tags per Movie                               words per Movie
             MoTags                5595                      85                           2.82                                         71
Table 2
Main statistics of the dataset.


metrics; finally, we show an ablation study aiming at highlight the benefits of the proposed
approach.
   Our approach has been evaluated on a novel media content dataset gathering different
types of information on a movie catalogue (e.g., movie plot, trailer, poster, synopsis, tags, etc.).
Specifically, we focused our analysis on text data (i.e., movie plots) and tags extracted by multiple
open sources such as Wikipedia, Wikimedia, and TMDB. When available, an extended plot is
associated with the movie otherwise it is replaced with its synopsis. As shown in Table 2, our
dataset contains ∼5k movies, with about 3 tags per movie. The complete list of tags (85) is
showed in Figure 4. It is important noting that a restricted number of tags (mainly the genres)
occurs more frequently than others, that can be considered as keywords summarizing some
aspects of the movie. In particular, we can see that the data distribution exhibits a long tail
shape.
   The experiments are perfomed on a DGX machine equipped with V100 GPUs. The dataset is
split in training and test set respectively with 70%/30% percentages. In more detail, the dataset
is partitioned in a stratified fashion so to reduce the sampling error. Adam is used as optimizer
while, as mentioned above, we exploits two loss functions to handle the imbalance problem i.e.,
the triplet loss and the CB-loss. The first one is used during the Embedder learning phase while
the last is used for training the local models. As regards the Clusterer, the number of groups 𝑘
has been empirically determined to 20. As a result of the clustering phase, each local model can
focus the learning on a limited number of tags, in particular the average number of tags per
cluster is ∼ 12.
   In Table 3 we report F1 score averaged according to two different strategies, respectively
named macro and micro [14]. In the former, the F1 is computed for each class and then it is
�                   3
              10
occurrences




                   2
              10
                                               scifi



                                           LGBT
                                        musical
                                           biopic
                                          drama
                                        comedy
                                           action
                                           thriller
                                            crime




                                       violence
                                       dystopic




                                           magic
                                           horror
                                      romantic
                                         fantasy




                                        mystery
                                               war
                                        wartime
                                        western
                                               noir
                                          sexual
                                   serial killer
                                         splatter
                                    animation
                                        aviation
                                     monsters


                                       futuristic
                                              love


                                           chase
                                           space
                                          history


                                           prison
                                          politics
                                           nudity
                                  teen movie

                                       revenge


                                   espionage
                                       neo-noir
                                  martial arts


                                      terrorism
                                       zombies
                                           family

                                suspenseful
                                     vampires
                                   time travel
                                         murder
                                fight scenes
                                             sport
                                        children
                                             alien
                                           music




                                          creepy


                                   small town
                                               cult

                                          justice
                        romantic comedy
                                    adventure
                            comedy drama




                              documentary

                               supernatural




                                   inbreeding




                                            death
                                          surreal

                                          pirates
                                 melodrama
                                        cartoon



                                        gunfight

                                    loneliness
                                         religion
                               erotic thriller
                            horror comedy




                              bank robbery




                            disaster movie
                             coming of age
                       police investigation
                                 high school



                                 parenthood
                                  super hero
                                     true story




                              death penalty

                         alcohol addiction
                                    friendship
                          action comedies
                                  prostitution
                               pornography
                                                    tag


Figure 4: Label distribution. There are 85 tags comprising genres (e.g.: Drama, Comedy) and more
specific keywords (e.g.: parenthood, small town). It shows a long-tail shape because of few tags (movie
genres) occurring most of the times.

                                 Method          Micro-F1   Macro-F1
                                 BERT            0.176      0.025
                                 BERT*           0.498      0.163
                                 𝑜𝑢𝑟_𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ    0.507      0.253
Table 3
Experimental results on movie tags prediction. BERT* denotes the version trained using Triplet Loss.


averaged, whereas in the latter, the cumulative sum of the counts of various true/false posi-
tive/negative is computed, and then the overall measure is calculated. While macro-averaging
weights all classes equally, micro-averaging favors bigger classes.
   The results shown in table 3 highlight the poor performance of the base model, i.e. the model
trained on all tags. It is unable to handle the high number of tags (i.e. classes) and provides
inaccurate predictions. The comparison of the values of Micro-F1 and Macro-F1 highlight the
influences of the majority classes such as Drama or Comedy on the overall performances. The
low value of Macro-F1 show that the model is unable to detect the under-represented tags which
are the majority in the dataset.
   The adoption of the triplet loss architecture allows for improving the performances of the base
model, although the low value of the Macro-F1 indicates poor performances on the minority
classes. Finally, the full approach, named in table as 𝑜𝑢𝑟_𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ, allows for improving also
the Macro-F1 value.


5. Conclusions and future work
Enriching metadata with informative labels is a crucial task for the enterprises operating in the
media content delivery field. However, automating this process requires to cope with different
challenging issues. In this work we proposed a hierarchical DL-Based approach for extreme
multi-label classification aiming at providing accurate predictions for movie tagging task. An
experimentation conducted on a real dataset demonstrates the quality of the approach.
  As a pointer of further research, we aim at boosting the overall performance of the proposed
�approach by integrating information coming from unlabeled data in a semi-supervised or self-
supervised way. Also, active learning schemes can be fruitfully exploited by implementing
ad-hoc oracle labeling strategies. Finally, we are interested to extend the experimentation for a
fully multi-modal scenario by including heterogeneous data e.g., movie posters and trailers.


Acknowledgments
This work was supported by PON I&C 2014-2020 FESR MISE, Catch 4.0.


References
 [1] Y. Le Cun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (2015) 436–444.
 [2] C. Wu, C. Wang, Y. Zhou, D. Wu, M. Chen, J. H. Wang, J. Qin, Exploiting user reviews for
     automatic movie tagging, Multimedia Tools and Applications 79 (2020) 11399–11419.
 [3] J. Arevalo, T. Solorio, M. Montes-y Gómez, F. A. González, Gated multimodal units for
     information fusion, arXiv preprint arXiv:1702.01992 (2017).
 [4] S. Kar, S. Maharjan, T. Solorio, Folksonomication: Predicting tags for movies from plot
     synopses using emotion flow encoded neural network, in: Proceedings of the 27th Inter-
     national Conference on Computational Linguistics, 2018, pp. 2879–2891.
 [5] J. Wehrmann, R. C. Barros, Movie genre classification: A multi-label approach based on
     convolutions through time, Applied Soft Computing 61 (2017) 973–982.
 [6] E. Fish, J. Weinbren, A. Gilbert, Rethinking movie genre classification with fine-grained
     semantic clustering, arXiv preprint arXiv:2012.02639 (2020).
 [7] J. Devlin, M. W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional
     transformers for language understanding, in: NAACL-HLT, Association for Computational
     Linguistics, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423.
 [8] M. KAYA, H. S. BILGE, Deep metric learning: A survey, Symmetry 11 (2019). doi:10.
     3390/sym11091066.
 [9] V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in:
     Proceedings of the 27th Int. Conf. on Machine Learning, ICML’10, 2010, pp. 807–814.
[10] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing
     internal covariate shift, in: Proc. of the 32Nd Int. Conf. on Machine Learning - Volume 37,
     ICML’15, 2015, pp. 448–456.
[11] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A
     simple way to prevent neural networks from overfitting, Journal of Machine Learning
     Research 15 (2014) 1929–1958.
[12] M. Guarascio, G. Manco, E. Ritacco, Deep learning, Encyclopedia of Bioinformatics and
     Computational Biology: ABC of Bioinformatics 1-3 (2018) 634–647.
[13] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, S. Belongie, Class-balanced loss based on effective number
     of samples, in: Proceedings of the IEEE/CVF conference on computer vision and pattern
     recognition, 2019, pp. 9268–9277.
[14] M. Sokolova, G. Lapalme, A systematic analysis of performance measures for classification
     tasks, Inf. Process. Manage. 45 (2009) 427–437.
�