A comparison of emotion annotation approaches for text
Wood, Ian McCrae, John Andryushechkin, Vladimir Buitelaar, Paul
Wood, Ian
McCrae, John
Andryushechkin, Vladimir
Buitelaar, Paul
Repository DOI
Publication Date
2018-05-11
Type
Article
Downloads
Citation
Wood, Ian; McCrae, John; Andryushechkin, Vladimir; Buitelaar, Paul (2018). A comparison of emotion annotation approaches for text. Information 9 (5),
Abstract
While the recognition of positive/negative sentiment in text is an established task with many standard data sets and well developed methodologies, the recognition of a more nuanced affect has received less attention: there are few publicly available annotated resources and there are a number of competing emotion representation schemes with as yet no clear approach to choose between them. To address this lack, we present a series of emotion annotation studies on tweets, providing methods for comparisons between annotation methods (relative vs. absolute) and between different representation schemes. We find improved annotator agreement with a relative annotation scheme (comparisons) on a dimensional emotion model over a categorical annotation scheme on Ekman's six basic emotions; however, when we compare inter-annotator agreement for comparisons with agreement for a rating scale annotation scheme (both with the same dimensional emotion model), we find improved inter-annotator agreement with rating scales, challenging a common belief that relative judgements are more reliable. To support these studies and as a contribution in itself, we further present a publicly available collection of 2019 tweets annotated with scores on each of four emotion dimensions: valence, arousal, dominance and surprise, following the emotion representation model identified by Fontaine et al. in 2007.
Funder
Publisher
MDPI AG
Publisher DOI
10.3390/info9050117
Rights
Attribution-NonCommercial-NoDerivs 3.0 Ireland