Tribe Dynamics

The fundamental goal of the project is to improve upon Tribe’s proprietary classification model, both from performance and operational standpoints to essentially develop a brand classification model that scales well across languages with limited amount of training data. Crucially, the model is expected to transfer knowledge learned from posts in one language when trying to understand posts in another language. To this extent, we investigated two approaches that could take on this challenging task. First, we trained multi-language aligned word embedding representations using neural language models from several sources of data, thus creating a universal compact language representation in the latent space. This space allows for any classifier to be built on it and will serve as a universal model for any language in future. Additionaly, we developed an evaluation framework of these word embeddings representation, through a web-based visualization tool that serves as a facilitator for Tribe’s developers to join the human-in-the-loop pipeline, to directly assess the quality of the representation and devise where to act in case something needs to be improved. Secondly, we develop another model which uses our core competency in English and translates posts word-by-word to English. In doing so, we also learn a mixture of translations(a graphical model where we do inference via Expectation Maximization) since translations are not on-to-one maps. With both these techniques, we were able to accomplish knowledge transfer, better maintenance because of single model, data efficiency(model performs well with a subset of data in target languages) and compact contextual representations. The project employs several salient concepts from natural language processing such as bag of words/n-grams, logistic regression, neural network language models, LSTMs, word embeddings, multi-channel embedding training, structured graphical models, exact inference via expectation maximization and general optimization problems in embeddings alignment. The classifiers built in these models are applicable far and wide beyond Tribe Dynamics' business problems in all settings that require scalable cross lingual text classification such as sentiment analysis, content classification, spam detection etc.

For our study, Tribe provided us 106 datasets (one dataset per brand) in which entries are specified as follows:

These datasets are used to serve the task of binomial classification, which means determining whether a particular social media post is about a specific brand or not. Each of the entries in the datasets represents a social media post that was gathered querying social media platforms, according to keywords provided by Tribe’s in-house experts. Most of these features are self explanatory. Following is a brief description of the less immediate ones:

In order to provide a more accurate understanding of the pipeline which we wanted to deploy, we decided to focus our attention only on English and Italian languages. First, because three components of our group are native Italian speakers, which simplifies the process of manual evaluation of the provided text. Furthermore, Tribe wants to expand internationally to Europe. Italian is a Latin language, so it shares with Spanish and French - two of the most common idioms of the European continent - the same linguistic structure, thus enabling the possibility of transfering the same ideas that were applicable in Italian, to the other mentioned languages.
It is important to notice that, on this reduced datasets, the disparity between the number of English training instances and Italian ones is substantial. Moreover, we identified a considerable class imbalance among all datasets, which greately impacts the performance of the classification and which forced us to use other performance metrics, instead of accuracy, to evaluate the results.

For the purpose of training the word embeddings, Tribe also provided us two large sized datasets (7’739’316 instances for the Italian dataset, 9’154’680 for the English one), which gathered a set of scraped social media posts, without any data preprocessing or labeling previously performed on them. The csv files simply look like this:

To make the computations feasible and reduce the amount of noise present in the data, we had to perform some data cleaning procedures, which we will detail in this upcoming chapter.

Before performing the cleaning of each post in the unlabeled datasets, we perform a sentence tokenization of the entire corpus, through the NTLK sentence tokenizer. This facilitates the cleaning process with regular expressions and avoids merging together multiple sentences, after punctuation is removed, which would result in an undesired behavior if we want to use this data for language modeling purposes.

Inspired by the preprocessing Perl script that was used to clean the training data for the FastText Wikipedia embeddings - which we will detail later on - each of the sentence tokenized posts is processed with the following instructions:

remove_regexp_from_string(input_string, regexp_str, replacement) using regular expressions Python’s package re, replaces each match of the regular expression in the input string, with the replacement parameter, eventually producing a cleaned string that can be used as a building block of the word embeddings training corpus.

The primary programming language that was used in this project is Python. To process the data, we took advantage of Pandas and NLTK. For statistical modeling, we used scikit-learn and PyTorch, as the deep learning framework to learn the embeddings, because of its strong GPU acceleration capabilities. The web-based visualization tool was built using HTML, CSS and Javascript, by means of its data visualization library D3. Amazon Web Services was used to train the embeddings on the cloud, taking advantage of an NVIDIA GK210GL (Tesla K80) GPU. Finally, GitHub was used for project collaboration and version control.

Before diving into the gory details of how we were able to train our embeddings, let's first define what word embeddings are. Word embeddings, by definition, is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with much higher dimension. The idea behind word embeddings is that words are clustered by meaning in this multi-dimensional vectorial space. A model - in our case a neural network - to learn this representation, it must rely on a fundamental linguistic assumption: words appearing in similar contexts are related to each other semantically, also known as the distributional hypothesis.

A language model is a probability distribution over sequences of words. Given such a sequence, say of length m, it assigns a probability to the whole sequence. In our case, we have trained a Recurrent Neural Network word-level language model. That is, we gave to the RNN - precisely an LSTM (Long Short-Term Memory) - a sequence of words, coming from a social media post, and asked it to model the probability distribution of the next word in the sequence given a sequence of previous words, repeating this process for each post in our training corpus. In terms of inputs and expected outputs, for each padded sequence - detailed in the previous chapter - the neural network takes as input all the words in the sequence, but the last one, while the target is all the words in the sequence, but the first one. This way, for each word in input, the neural network tries to predict the next word in the sentence in output. The reason why this model is suitable for learning word embeddings is the argument that if the model is able to learn a language (able to predict next word), its hidden layers should be able to map the context behind the words. Hence, we use the hidden layer vectors as our embeddings.

As a matter of fact, because our primary task remains to learn this word embedding representation, words are fed into the just mentioned hidden layer, which is able to learn - through training - a representation, for these words, into a multi-dimensional vector space. The output of this hidden layer acts as the input for the remaining part of the model, which is an LSTM, that represents the recurrent neural network which is able to tune its weights in order to achieve the task of language modeling. Through this pipeline, word embeddings are learnt along the way, with the expectation that, once the training is over, the model was able to encode, in a vectorial representation, the semantic information of each word encountered in the training corpus.

The reason why we used 300 dimensional word embeddings is because we have decided to enhance the quality of our representation with state-of-the-art embeddings coming from FastText, a library for efficient learning of word representations and sentence classification, developed at Facebook.

To do such enhancement, we performed a customization of our neural network setting. Precisely, we turned it into a multi-channel neural network. The idea behind multi-channel neural networks is to split the network in two parts - known as static and dynamic channels - and perform backpropagation only on the dynamic channel, thus keeping unchanged the parameters of the static one. One would think, why do such thing? The purpose of deploying a multi-channel neural network is that we wanted to levearge the quality of pre-trained embeddings - learned on Wikipedia text - that are already able to accurately model words with their semantic information, and adapt this representation to the fashion and cosmetics domain. This way, we expect our embeddings to perform better once used into Tribe's classification tasks.

To adapt this concept to our framework, we set up two embedding layers:

Once these word embeddings have been learned separately, both in English and Italian, they are ready to be aligned in the same vector space.

Monolingual word vectors embed language in a high-dimensional vector space, such that the similarity of two words is defined by their proximity in this space. They enable us to train sophisticated classifiers but they require independent models to be trained for each language. Crucially, training text obtained in one language cannot improve the performance of classifiers trained in another, unless the text is explicitly translated. Because of this, increasing interest is now focused on bilingual vectors, in which words are aligned by their meaning, irrespective of the language of origin. The idea is that, starting from two sets of word vectors in different languages - in our case, the previously computed English and Italian embeddings - we learn a linear matrix , trained using a dictionary of shared words between the two languages, to then map word vectors from the "source" language into the "target" language.

As stated above, our method requires a training dictionary of paired vectors, which is used to infer the linear mapping . Typically this dictionary is obtained by translating common source words into the target language using Google Translate, which was constructed using expert human knowledge. However most European languages share a large number of words composed of identical character strings (e.g. words like "Boston", "DNA", "pizza", etc.). It is probable that identical strings across two languages share similar meanings. Following this reasoning, we extracted these strings to form a "pseudo-dictionary", compiled without any expert bilingual knowledge, containing words that appeared both in the English vocabulary and the Italian vocabulary. We call such words anchor words.

Since euclidean distance is a direct measure of similarity in these mono-lingual embeddings, we want to align embeddings of two languages only using isometric maps, i.e. functions that preserve the relative distance between words. For instance, rotation is one such transformation where when we rotate the entire embedding cloud about any axis, we do not change the inter-word distances. We formally define this rotation using an orthonormal matrix transformation (remember orthonormal transformation can rotate the matrix and cannot skew it). If is a set of source anchor word embeddings and is a set of target anchor word embeddings, then we can define this problem as follows where we minimize the norm of difference between the transformed and target embeddings:

This problem is famously called the Orthogonal Procrustes Problem which has a neat closed form solution that is just a product of two rotation matrices of a singular value decomposition. Remember that singular value decomposition gives a product of three matrices with , being rotation matrices and being the skew matrix. Since our solution is a product of two rotation only matrices, it is in essence a rotation matrix only. Thus, we have achieved the closest possible alignment between the two embeddings without any skewing of inter-word distances in either language.

The overall procedure is outlined in the following "pythonic" pseudocode:

It is important to notice that in our case, we performed all of the computations moving from English to Italian, which means that we have always applied the linear transformation to the English embeddings matrix. Theoretically, given that some words in English can translate to either male or female form of the Italian word, this should always be the approach to follow.

In this chapter, we outline all the parameters that we have set to perform our evaluation.

We encountered severe limitations due to to the computing power needed to perform an appropriate training, which forced us to reduce the amount of training instances and the size of the batches, in order for us to come up with a feasible training time for each epoch (i.e. about 1 hour each, for both Italian and English embeddings)

One of the primary objective of this project was to build a human-in-the-loop pipeline for training a foreign language text classifier. According to Wikipedia, human-in-the-loop is defined as a model that requires human interaction. To accomplish this task, we decided to set up a visualization tool to visualize word embeddings, providing to the user the capability to actively evaluate them.