Aggressive Language Identification Using Word Embeddings and Sentiment Features

This notebook presents the code of my participation in the First Shared Task on Aggression Identification. More details about the approach can be found in:

Constantin Orǎsan (2018) Aggressive Language Identification Using Word Embeddings and Sentiment Features. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), p. 113 - 119, Santa Fe, USA, August, 25, http://aclweb.org/anthology/W18-4414

Abstract

This paper describes our participation in the First Shared Task on Aggression Identification. The method proposed relies on machine learning to identify social media texts which contain aggression. The main features employed by our method are information extracted from word embeddings and the output of a sentiment analyser. Several machine learning methods and different combinations of features were tried. The official submissions used Support Vector Machines and Random Forests. The official evaluation showed that for texts similar to the ones in the training dataset Random Forests work best, whilst for texts which are different SVMs are a better choice. The evaluation also showed that despite its simplicity the method performs well when compared with more elaborated methods.

The first step is to import all the necessary modules:

%%capture --no-display
# the previous line is to suppress warning caused by using deprecated functions in numpy
# in this case it's safe to ignore them

import csv
import re
import emoji
import nltk
import math
import numpy as np
from nltk.tokenize import word_tokenize
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score

Initialise various variables

# debug is used to supress various warnings and debugging messages
debug = False

# path to training and dev data. You can obtain the data from the workshop organisers. 
path_to_data = "english/"

The purpose of the task was to classify short texts into three categories Overtly Aggressive (OAG), Covertly Aggressive (CAG) and Non-aggressive (NAG).

labels = ['NAG', 'CAG', 'OAG']

Step 1: Load the data

def read_data(file_name, test_data=0):
    '''
    Reads the data from a CVS file. Each instance is stored in a tuple with the 
    structure (ID, text, label)
    Returns a list which contains all the instances from the file
    '''
    data = []
    with open(file_name, encoding = 'utf8') as csvfile:
        reader = csv.reader(csvfile)
        for row in reader:
            if test_data:
                id_ex, text = tuple(row)
                data.append([id_ex, text, "UNK"])
            else:
                id_ex, text, label = tuple(row)
                data.append([id_ex, text, label])
            
    return data

print("Reading the data ...", end="")
training = read_data(path_to_data + 'agr_en_train.csv')
dev = read_data(path_to_data + 'agr_en_dev.csv')
print("done!")

Reading the data ...done!

The data is now ready to be used.

# the second instance from the training data
print(training[1])

['facebook_corpus_msr_466073', "Most of Private Banks ATM's Like HDFC, ICICI etc are out of cash. Only Public sector bank's ATM working", 'NAG']

We can have a look at how the instances are distributed between different classes.

%%capture --no-display
%matplotlib inline

import matplotlib.pyplot as plt
from collections import Counter
from pandas import DataFrame 

def plot_data(df):
    ax = df.plot(kind='bar')
    total = 0
    for p in ax.patches:
        total += p.get_height()
    for p in ax.patches:
        ax.annotate(np.round(p.get_height(),decimals=2), (p.get_x()+p.get_width()/2, p.get_height()/2), 
                    ha='center', va='center', xytext=(0, 10), textcoords='offset points', 
                    bbox=dict(facecolor='yellow', alpha=0.5))
        
        ax.annotate(str(np.round(p.get_height()/total * 100,decimals=2)) + "%", (p.get_x()+p.get_width()/2, p.get_height()/2), 
                    ha='center', va='center', xytext=(0, -20), textcoords='offset points', 
                    bbox=dict(facecolor='gray', alpha=0.5))

# plot the training data
df = DataFrame.from_dict(Counter([instance[2] for instance in training]), orient='index')
df.rename({0: "Training"}, axis='columns', inplace=True)
plot_data(df)

# plot the dev data
df = DataFrame.from_dict(Counter([instance[2] for instance in dev]), orient='index')
df.rename({0: "Dev"}, axis='columns', inplace=True)
plot_data(df)

Step 2: Calculate the features

2.1 Features based on word embeddings

The main purpose of this method was to explore to what extent word embeddings can be used as features for traditional machine learning methods. For the experiments in the paper, I used the pretrained GloVe vectors from https://nlp.stanford.edu/projects/glove/

The experiments were run using glove.840B.300d.zip (vectors generated using 840B tokens from Common Crawl, 2.2M vocabulary size, cased, 300d vectors)

# path to the uncompressed GloVe vector
path_to_glove="/home/dinel/Projects/2018-aggression/glove/glove.840B.300d.txt"

def loadGloveModel(gloveFile):
    '''
    Loads the embeddings into a dictionary. This will consume lots of memory.
    Returns the dictionary where the keys are the words and the values the embeddings
    '''
    
    print("Loading Glove Model")
    f = open(gloveFile,'r')
    model = {}
    for line in f:
        splitLine = line.split()
        word = splitLine[0]
        try:
            embedding = np.array([float(val) for val in splitLine[1:]])            
            model[word] = embedding
        except:
            if debug: print("Problem:", line)
    print("Done.",len(model)," words loaded!")
    return model

# load the embeddings
embeddings = loadGloveModel(path_to_glove)

Loading Glove Model
Done. 2195884  words loaded!

# we can check the vector corresponding to a word
print(embeddings["hate"])

[-6.1052e-01  1.1656e-01 -5.0648e-01 -3.2216e-01 -9.9742e-02  1.0182e-01
  3.1042e-01 -1.8155e-01  3.1774e-01  2.1537e+00 -8.1129e-02 -3.8734e-01
  6.8008e-02 -5.8029e-03 -5.3814e-01 -9.8787e-02 -8.4160e-02  3.1755e-02
 -2.9811e-01  5.0496e-01  4.1719e-01  1.1462e-01  2.5330e-01 -1.6668e-01
  2.6645e-02  4.5709e-02 -5.9685e-01 -3.9645e-01  4.3089e-01  1.3014e-01
 -4.0591e-02  6.6024e-01 -5.3007e-01 -8.1629e-02  3.3484e-01  1.6258e-01
  3.9029e-02  7.2786e-02 -1.8635e-01  3.8104e-01 -5.0557e-01 -3.0910e-02
 -3.2578e-01  3.8818e-01  1.4605e-01  3.8982e-02 -1.4992e-01  6.0316e-01
 -3.0801e-02  1.3456e-01 -6.2982e-01  5.8746e-02  1.5878e-02 -7.8831e-02
  1.4754e-01  5.2191e-02 -4.3654e-01 -1.6707e-01 -1.5184e-02 -1.6073e-01
 -2.1243e-02  8.3719e-02 -6.5062e-01 -7.9566e-02 -4.5479e-02 -1.0070e-01
 -1.4044e-01 -4.1403e-02  1.2033e-01  2.6015e-03  1.0318e-02  1.2444e-01
  3.0785e-01 -2.4270e-01  4.9056e-02  2.1891e-01  1.5346e-01  1.6682e-01
  5.0732e-01  5.9128e-01 -3.0452e-01 -5.8077e-02  1.4405e-01 -4.4923e-01
  2.3952e-01 -1.8347e-01  2.6224e-01 -7.6430e-01  1.1990e-01 -4.5683e-01
 -1.4176e-01 -1.6581e-01  1.2650e-01  2.3494e-01  2.1847e-01  1.2909e-01
  4.9598e-02  4.8204e-01 -1.7446e-02 -3.5697e-01  3.1545e-01 -2.6209e-01
 -3.4744e-02 -4.2803e-01  5.4802e-01 -7.5777e-01 -1.6709e-01 -2.3587e-02
 -2.7345e-01  9.4488e-02 -9.8469e-02 -2.5598e-01  2.9519e-02 -4.2601e-01
 -2.5830e-01 -1.1096e-02  1.1266e-01 -5.7712e-01 -1.4331e-01 -4.8758e-02
  8.4046e-02 -1.4820e-01  6.3360e-02 -5.2884e-01  5.1334e-01  5.5074e-02
  3.3447e-01 -4.5821e-02 -8.4835e-02  5.3418e-02 -6.0270e-01  1.0075e-01
  3.4882e-02  1.3661e-01 -5.7156e-02 -3.2932e-01 -5.0855e-01 -1.0988e-01
  8.5876e-02 -1.2094e-01 -2.9525e+00 -9.1559e-02  4.1377e-01  2.9983e-02
 -4.5670e-01  3.1407e-01 -2.9968e-01  4.8521e-02 -5.2216e-01 -6.1831e-01
  3.2831e-01  2.2728e-01  2.7956e-01  1.3687e-01 -3.8543e-02  2.9034e-01
 -1.5775e-01 -6.1231e-01 -4.9361e-02  1.4326e-01 -2.3045e-01  5.2664e-02
  9.3194e-02  2.7935e-01 -1.8719e-01  4.2922e-01 -1.2092e-01 -3.3848e-02
 -6.7490e-01 -4.7605e-01 -3.0468e-01 -2.6370e-01  2.5106e-03 -4.6804e-01
  6.8331e-01  1.6012e-01 -5.1522e-01  1.8021e-01  2.5192e-01 -3.2158e-01
 -2.5619e-02 -3.0824e-01 -6.2287e-01  8.0491e-02 -2.8676e-02  2.4823e-01
  6.2337e-02 -4.3325e-01  5.4157e-01  1.4867e-01  2.3373e-01 -4.6265e-01
 -2.9033e-01 -2.1299e-01  3.3593e-01  6.8914e-02  2.4089e-01 -3.2855e-01
 -1.5312e-01  3.4413e-01  3.3486e-02 -5.7180e-02  9.8070e-02 -4.4647e-02
  3.5703e-01 -3.5915e-01  7.0520e-02  7.5669e-02  1.6122e-01  1.5627e-01
 -8.8761e-02  5.2585e-01  2.7204e-01  3.4040e-01  7.6718e-02  5.6750e-03
 -2.9162e-01 -1.6033e-01 -3.9822e-01 -3.8824e-02  8.5246e-03  4.1401e-02
 -2.2201e-01 -2.9588e-01 -7.4356e-02  6.3065e-02  5.1414e-02 -1.1666e-01
 -2.0297e-01 -3.3333e-01 -5.7485e-01  1.0103e-01 -1.3429e-01 -5.7319e-02
 -7.3142e-02 -1.7734e-01  6.7575e-01 -7.7484e-02  2.9055e-01  3.8863e-01
  5.1169e-02  3.3529e-01  1.1840e-01 -7.4655e-03 -2.2907e-01  2.0701e-01
 -2.4925e-01  5.5953e-02 -1.5499e-01  5.6069e-01 -6.4385e-02 -3.0198e-02
 -7.5703e-02 -1.9910e-01 -2.2604e-01 -3.7639e-02 -2.0899e-01 -8.6741e-02
 -2.2423e-01  1.9680e-01  5.7717e-02  3.1425e-01 -1.1041e-01  3.5558e-01
  1.7370e-01  3.0618e-01 -6.7071e-02  1.8406e-01  6.8487e-01  4.1663e-01
  3.0946e-02 -5.0516e-01 -2.8229e-01 -2.8282e-01  2.3254e-01 -2.1508e-01
 -8.4610e-02 -3.9999e-02 -3.1937e-01  6.1145e-02  1.9686e-01 -1.1733e-01
 -4.5938e-02 -1.4098e-01 -1.0227e-01  3.4548e-01  5.5819e-01 -3.7241e-01
 -9.7569e-02 -3.4458e-01  1.7395e-01 -1.4566e-02 -2.8684e-01 -4.1603e-01
 -1.0828e-02  4.0514e-01 -3.4146e-01  1.0321e-01 -8.8201e-02  6.7543e-01]

The features corresponding to a text are calculated by averaging the vectors corresponding to the tokens contained it the text. The function below can be used to find out which tokens are not present in the embeddings. The input of the function is a list which contains all the instances. The text is in the second position of an instance. The tokenizer from NLTK is used.

def calculate_vector_repr(data):
    '''
    Calculates the vector representation of the text. The input of the function 
    is a list which contains the input (training or testing). The function modifies
    this data by adding the representation at the end of each instance.
    '''
    
    if debug: not_found = open("not-found.txt", "w")
    for instance in data:
        text = instance[1]

        sentVector = np.zeros(300)
        counter = 0

        for word in word_tokenize(text):        
            try:
                sentVector += embeddings[word.lower()]
                counter += 1
            except:
                if debug: not_found.write(word + "\n")
                    
        if counter: 
            sentVector = sentVector / counter
        instance.append(sentVector)
        
    if debug: not_found.close()

If debug=True calling calculate_vector_repr creates/appends to a file called not-found.txt all the words that are not present in the embeddings. This is very useful for measuring the coverage of the word embeddings. In the case of this task, most of the non found words are either poorly tokenised words or words they are not English.

data = [["dummy", "this is a test"]]
calculate_vector_repr(data)
data

[['dummy',
  'this is a test',
  array([-1.69022500e-02,  2.87252250e-01,  1.86550750e-02,  1.34868000e-01,
          4.24300000e-02, -2.67570750e-01,  2.58480000e-02, -2.83737750e-01,
          5.71710000e-02,  2.15382500e+00, -1.57307175e-01, -5.59105000e-02,
         -2.40000000e-03,  4.53894500e-02, -9.60605000e-02, -1.85047625e-01,
          1.54570000e-02,  1.65845000e+00, -1.73122500e-01, -3.72495000e-02,
         -2.85275000e-02, -2.71057750e-01, -4.75250000e-02,  1.30351075e-01,
          6.37230000e-02,  2.03934500e-01,  3.63950000e-03, -1.35885500e-01,
          4.68250000e-03, -1.42760000e-02, -2.46225000e-04,  8.99967500e-02,
          3.52918750e-01, -1.09063000e-01,  4.67625000e-02,  4.68908400e-02,
          6.66356250e-02, -6.19975000e-02, -1.38557592e-01, -3.94212250e-01,
          2.24767500e-01,  1.93722500e-01,  1.34370750e-01, -9.73465000e-02,
          5.74125000e-02,  4.60950000e-03, -2.31309000e-01,  4.35100000e-02,
         -1.65817500e-01,  8.05360750e-02, -1.69070000e-01,  2.49950000e-01,
         -1.23277500e-02, -7.32342500e-02,  4.79375000e-02,  1.22529500e-01,
          8.54340750e-02,  7.20050000e-03,  1.58414650e-01, -2.26668750e-01,
         -4.65070500e-02,  1.57557250e-01, -8.18155000e-02,  3.33497000e-01,
          3.04625250e-01, -1.29667750e-01, -6.98820000e-02, -1.36443000e-01,
         -5.63567500e-02,  2.13260000e-01,  4.11295000e-02,  1.87477500e-02,
          2.10122500e-01,  2.84410000e-02, -2.26245000e-02, -9.80845000e-02,
          3.33040000e-02,  3.45000000e-05, -2.37012500e-01,  4.54745000e-01,
         -3.48975000e-02, -1.04009750e-01,  6.17780000e-02,  2.36718587e-01,
          7.68100000e-02, -2.09564500e-01, -2.31786000e-01, -1.67525500e-01,
          1.68050000e-01, -1.97675000e-02, -8.27067500e-02,  4.58715000e-01,
         -6.05664000e-02, -3.15750000e-02,  6.39547500e-02,  9.35362500e-02,
         -5.18382500e-02, -3.98707500e-02,  2.06635250e-01,  2.85175000e-01,
          1.55960000e-01,  4.57625000e-02, -1.07550000e-01, -9.50000000e-05,
          1.12489250e-01, -8.25805000e-01,  1.29028750e-01,  1.04883500e-01,
         -1.27268000e-01, -7.81945000e-02,  7.36750000e-03, -7.81790000e-02,
         -2.02637500e-02, -1.34450000e-03, -8.54400000e-03,  1.56997500e-01,
         -1.45537500e-01,  3.73980000e-01,  1.38417500e-01, -4.05875000e-02,
         -7.90592500e-02, -1.69187250e-01,  3.08477500e-02, -1.67063000e-01,
         -7.95925000e-03,  1.61058750e-01, -2.04527500e-01, -2.07422500e-01,
         -1.27421250e-02, -7.25512500e-02, -9.06920000e-02, -2.69842500e-02,
         -2.09380250e-01,  2.13083500e-01,  1.25434275e-01, -1.76262145e-01,
          7.63952500e-02,  3.06730000e-02,  8.89172500e-02, -1.02119500e-01,
         -1.19806000e+00,  5.81725000e-02,  1.54513500e-01,  7.77572500e-02,
         -1.69317500e-02, -3.48612500e-01, -9.28835000e-02,  3.63925000e-02,
          3.26028000e-01, -7.05635000e-02, -6.92765000e-02,  1.33272500e-01,
          8.09050000e-02,  1.75165000e-02,  1.17953450e-01,  5.93287500e-02,
         -3.31965000e-01,  1.27641750e-01, -8.66400000e-02, -2.57092500e-01,
          3.08382500e-02,  2.95660000e-02,  6.91310000e-02, -3.31830000e-02,
         -1.73564100e-01, -2.63792500e-01, -8.19967500e-02,  1.17067000e-02,
          2.03666773e-01, -6.23300000e-02,  3.38693750e-02, -1.43808000e-01,
          1.68575000e-01, -1.03225000e-01, -4.41947500e-01,  4.37812500e-02,
         -2.58666250e-02, -9.01300000e-02, -3.02050000e-02,  2.95564750e-01,
         -6.80797500e-02, -1.03582500e-01, -1.03924750e-01, -1.36811000e-01,
          2.03645000e-02, -1.94859405e-01, -1.80158000e-01,  2.78492500e-01,
         -2.19137625e-01, -3.88630000e-02, -2.44470000e-01, -1.10500000e-04,
         -5.46747500e-01,  1.68005000e-01,  2.68930000e-01,  2.48715000e-01,
         -1.72146250e-01, -9.41317500e-02,  2.36864000e-01, -5.34050000e-02,
         -8.16077500e-02,  1.42102250e-02, -3.43040000e-01,  1.20894750e-01,
          1.29999250e-01, -5.10635000e-02, -4.50082500e-02, -4.19900250e-02,
          1.44472500e-01,  1.34179750e-01, -2.48782500e-01, -8.66902500e-02,
          5.88647500e-02, -2.94300000e-01,  5.68540000e-02,  3.07725000e-01,
         -4.53325000e-02,  3.26032500e-01, -8.12095000e-02,  1.93018000e-01,
          8.21810000e-02, -7.04255000e-02, -6.08912500e-02, -1.65200000e-02,
          4.58710000e-02, -1.36672500e-01, -8.41520000e-02,  1.71415000e-01,
         -3.06767500e-02, -1.20317250e-01, -5.10400000e-02,  3.44682500e-01,
          3.89110000e-02,  1.09950000e-03, -1.77381538e-01, -2.05941125e-01,
          3.30971250e-02, -2.76257500e-02, -1.59756000e-01,  2.36192500e-01,
          1.62469250e-01, -9.33980000e-02,  1.74687750e-01,  4.63500000e-02,
          1.10741000e-01, -1.11376750e-01,  1.30968000e-01, -2.65440000e-01,
         -1.22173250e-01, -3.38005000e-02, -6.52402500e-02, -1.30703250e-01,
          3.08882500e-02, -5.88427500e-02,  4.71567500e-02,  1.22072250e-01,
          3.13256000e-01, -1.90145000e-01,  6.34067250e-02, -5.39375000e-03,
          1.88715250e-01,  2.44780000e-01,  1.06738275e-01, -1.66912250e-01,
          3.55732500e-02, -1.25672500e-02,  9.57420000e-02,  2.99566400e-01,
          6.62767500e-01,  8.87952500e-03,  3.39099250e-01,  2.39295000e-02,
         -1.02379000e-01, -1.54405000e-01, -7.59370000e-02, -3.98902500e-02,
         -4.68650000e-02, -3.52055000e-02,  1.60293175e-01,  2.04122750e-01,
          2.25618750e-01,  2.71627500e-02, -5.58942250e-02, -1.96019250e-01,
         -1.39595850e-01, -2.18355000e-01,  7.11065000e-02, -1.21249750e-01,
          2.16370000e-01, -2.25518250e-01, -8.21050000e-03,  1.00616350e-01,
          2.49310000e-02, -2.06725250e-01,  1.61813250e-01, -7.74647500e-02,
         -5.41150000e-03, -8.29525000e-02, -3.95275000e-02,  1.47072925e-01])]]

2.2 Extract sentiment features

The sentiment features are determined using SentiStrength which is able to assign each text a positive and a negative score. In order to facilitate processing, the input files were split into individual files, each containing only one text. The name of each file matched the ID of the text. The following function was used:

def splitData(file_name, train_data = True):
    with open(file_name, encoding = 'utf8') as csvfile:
        reader = csv.reader(csvfile)
        for row in reader:
            if train_data:
                id_ex, text, label = tuple(row)
            else:
                id_ex, text = tuple(row)
            
            f = open("split-test/" + id_ex, "w")
            f.write(text)
            f.close()
            

A shell script was run to generate the files containing the scores

for f in `find $1 -type f`
do
    cat $f | java -jar SentiStrengthCom.jar sentidata ./SentStrength_Data/ stdin > $f.senti
done

After this each file contained two numbers separated by a tab which represented the positive and negative scores.

path_to_sentiment_feats = "senti_feats/"

def getSentiFeatures(instance, path):
    '''
    Read the sentiment features corresponding to an instance. 
    Returns a list which contains two elements [positive score, negative score]
    '''
    f = open(path + instance[0] + ".senti")
    line = f.readline()
    f.close()
    return [int(x) for x in line.strip().split()]

2.3 Calculate the features based on emojis

The assumption is that some emoticons are more likely to indicate a type of aggression or the lack of aggression than others. The emoticons are converted to their corresponding strings description using emoji library.

def getEmojis(text):
    """
    Identifies all the emoji in the text using emoji library
    """
    return re.findall("(:[0-9a-zA-Z_]+:)", emoji.demojize(text))

The score of each emoji is calculating using TF*IDF with respect to each class. The implementation used in the official submission considered there are 4 classes instead of 3. This was done in order to avoid having a large number of emojis with score 0, which in turn proved to have a negative influence on the evaluation on the dev data.

def getEmojiScores(data):
    emos_dist = {'NAG':[], 'CAG':[], 'OAG':[]}
    emo_tfidf = {'NAG':[], 'CAG':[], 'OAG':[]}
    
    for instance in data:
        emos = getEmojis(instance[1])
        for emo in emos:
            emos_dist[instance[2]].append(emo)        
    
    for key,value in emos_dist.items():
        N = len(value)
        for emo,f in nltk.FreqDist(value).most_common(50):
            tf = f/N
            idf = (emo in emos_dist['NAG']) + (emo in emos_dist['CAG']) + (emo in emos_dist['OAG'])
            emo_tfidf[key].append((tf * math.log(4/idf), emo))
        
    return {label : {v:k for k, v in emo_tfidf[label]} for label in labels}

We can see which emojis are ranked in top 10 for each class.

from operator import itemgetter

emoji_scores = getEmojiScores(training)
for label in labels:
    print("\nLabel:", label)
    for em in sorted(emoji_scores[label].items(), key=itemgetter(1), reverse=True)[:10]:
        print(emoji.emojize(em[0]), em[0], em[1])

Label: NAG
♥ :heart_suit: 0.10836526343965343
❌ :cross_mark: 0.07126724532517748
💢 :anger_symbol: 0.048813181729573614
😂 :face_with_tears_of_joy: 0.0362641485696259
😯 :hushed_face: 0.018549009057237973
🌟 :glowing_star: 0.017572745422646502
😽 :kissing_cat_face: 0.015620218153463556
🏨 :hotel: 0.013667690884280612
🏯 :Japanese_castle: 0.013667690884280612
⭐ :white_medium_star: 0.012691427249689139

Label: CAG
😂 :face_with_tears_of_joy: 0.14426913454799128
😀 :grinning_face: 0.017123932884034573
😈 :smiling_face_with_horns: 0.009283221168213552
👏 :clapping_hands: 0.009283221168213552
😝 :squinting_face_with_tongue: 0.008251752149523157
🌹 :rose: 0.008251752149523157
👌 :OK_hand: 0.008251752149523157
😜 :winking_face_with_tongue: 0.007705769797815558
📢 :loudspeaker: 0.006188814112142368
🇺🇸 :United_States: 0.006188814112142368

Label: OAG
😂 :face_with_tears_of_joy: 0.108787338322102
🐣 :hatching_chick: 0.062130839714056724
❌ :cross_mark: 0.05436448474979963
😡 :pouting_face: 0.026592460478736046
🃏 :joker: 0.015532709928514181
🌝 :full_moon_face: 0.013591121187449907
💧 :droplet: 0.011649532446385634
😉 :winking_face: 0.008058321357192741
😏 :smirking_face: 0.008058321357192741
😑 :expressionless_face: 0.008058321357192741

For each new text we calculate three features, each corresponding to one of the classes to be predicted. The values of the features are the sum of the emoticons scores appearing in the text for the corresponding class. The values of these three features are normalised by the number of emoticons in the text.

def getEmoFeatures(text, scores):
    '''
    Returns the features corresponding to the emoji scores
    '''
    emos_features = {'NAG':0, 'CAG':0, 'OAG':0}
    emos = getEmojis(text)
    for emo in emos:
        for label in labels:
            emos_features[label] += scores[label].get(emo, 0)
            
    for label in labels:
        if len(emos): emos_features[label]/= len(emos)
        
    return emos_features

The examples below show the features on a few examples.

print(training[289])

['facebook_corpus_msr_429182', 'So humble you are Mukesh Ambani.. your  pieces about Entrepreneurs  should inspire a generation.. 🙏👍👌', 'NAG']

print(getEmoFeatures(training[289][1], emoji_scores))

{'NAG': 0.007278423690762763, 'CAG': 0.0051764745417459505, 'OAG': 0.0018802749833449726}

print(training[222])

['facebook_corpus_msr_1689985', 'Stupid survey 👎👎👎', 'OAG']

print(getEmoFeatures(training[222][1], emoji_scores))

{'NAG': 0.0, 'CAG': 0.0, 'OAG': 0.005640824950034919}

Train the algorithms

As stated above, the purpose of this method was to investigate whether we can use word embeddings (together with some other features) as features for traditional machine learning algorithms. The data for the ML algorithms is prepared by the extract_XY function. The implementation assumes that the emoji_scores and path_to_sentiment_feats variables are correctly initialised in the global scope.

def extract_XY(data, with_sentiment = 0, with_emojis = 0):
    '''
    Function which prepares the vectors for the machine learning algorithms
    The parameters with_sentiment and with_emojis control whether those
    features will be included
    Returns X and Y
    '''
    X = []
    Y = []
    for instance in data:
        features = list(instance[3])
        if with_emojis:
            features.extend(list(getEmoFeatures(instance[1], emoji_scores).values()))
        if with_sentiment:
            features.extend(getSentiFeatures(instance, path_to_sentiment_feats))
            
        X.append(features)
        Y.append(instance[2])
        
    return X, Y

The following two functions return the model produced by the machine learning algorithms from the training data provided as parameters. The two ML algorithms tried are Support Vector Machine and Random Forest. Each function takes parameters which can tune the performance of the learning algorithm. The values used in the experiments below are the best values determined through a grid search.

def trainSVM(X_train, y_train, c, gamma):
    print("Learning ...", end="")
    model = SVC(random_state=57, kernel='rbf', C=c, gamma=gamma)
    model.fit(X_train, y_train)
    print("done!")
    
    return model

def trainRandomForest(X_train, y_train, n_estimators):
    print("Learning ...", end="")
    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=10) 
    model.fit(X_train, y_train)
    print("done!")
    
    return model

Before we can start experimenting with the ML algorithms we need to calculate the vector representation for the texts. This will modify the training and dev variables. The main reason for implementing the method this way is because deriving the vector representation is time consuming and it shouldn’t be done in the extractXY function.

print("Calculating representation ... ", end="")
calculate_vector_repr(training)
calculate_vector_repr(dev)
print("done!")

Calculating representation ... done!

Use only word embeddings (not reported in the paper)

print("Calculating features ...", end="")
X_train, y_train = extract_XY(training)
X_dev, y_dev = extract_XY(dev)
print("done!")

Calculating features ...done!

SVM, c=2, gamma=0.5

c = 2
gamma = 0.5
model = trainSVM(X_train, y_train, c, gamma)
predicted = model.predict(X_dev)
print(f"SVM with c={c} and gamma={gamma}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
SVM with c=2 and gamma=0.5
accuracy= 0.5784738420526491
weighted f-score 0.574230558765779
Confusion matrix
 [[587 324 146]
 [304 863  66]
 [283 142 286]]

Random Forest, n_estimators=180

n_estimators = 180
model = trainRandomForest(X_train, y_train, n_estimators)
predicted = model.predict(X_dev)
print(f"Random forest with n_estimators={n_estimators}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
Random forest with n_estimators=180
accuracy= 0.5404865044985006
weighted f-score 0.5132813625882178
Confusion matrix
 [[581 425  51]
 [282 917  34]
 [408 179 124]]

Use word embeddings and emoji scores

print("Calculating features ...", end="")
X_train, y_train = extract_XY(training, with_emojis = True)
X_dev, y_dev = extract_XY(dev, with_emojis = True)
print("done!")

Calculating features ...done!

SVM, c=2, gamma=0.5

c = 2
gamma = 0.5
model = trainSVM(X_train, y_train, c, gamma)
predicted = model.predict(X_dev)
print(f"SVM with c={c} and gamma={gamma}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
SVM with c=2 and gamma=0.5
accuracy= 0.5768077307564146
weighted f-score 0.5723505370388843
Confusion matrix
 [[585 328 144]
 [302 863  68]
 [286 142 283]]

Random Forest, n_estimators=180

n_estimators = 180
model = trainRandomForest(X_train, y_train, n_estimators)
predicted = model.predict(X_dev)
print(f"Random forest with n_estimators={n_estimators}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
Random forest with n_estimators=180
accuracy= 0.5341552815728091
weighted f-score 0.5079277755736931
Confusion matrix
 [[560 434  63]
 [279 916  38]
 [400 184 127]]

Use word embeddings, emoji scores and sentiment scores

print("Calculating features ...", end="")
X_train, y_train = extract_XY(training, with_emojis = True, with_sentiment = True)
X_dev, y_dev = extract_XY(dev, with_emojis = True, with_sentiment = True)
print("done!")

Calculating features ...done!

SVM, c=5, gamma=0.5 (the best performing setting on unseen Twitter testing data)

c = 5
gamma = 0.5
model = trainSVM(X_train, y_train, c, gamma)
predicted = model.predict(X_dev)
print(f"SVM with c={c} and gamma={gamma}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
SVM with c=5 and gamma=0.5
accuracy= 0.5591469510163279
weighted f-score 0.5545043639597993
Confusion matrix
 [[533 352 172]
 [302 859  72]
 [272 153 286]]

Random Forest, n_estimators=160 (the best performing setting on unseen Facebook testing data)

n_estimators = 160
model = trainRandomForest(X_train, y_train, n_estimators)
predicted = model.predict(X_dev)
print(f"Random forest with n_estimators={n_estimators}")
print("accuracy=", np.mean(predicted == y_dev))
print("weighted f-score", f1_score(y_dev, predicted, average='weighted'))
print("Confusion matrix\n", confusion_matrix(y_dev, predicted))

Learning ...done!
Random forest with n_estimators=160
accuracy= 0.5441519493502166
weighted f-score 0.5214229496782077
Confusion matrix
 [[568 431  58]
 [283 919  31]
 [405 160 146]]