Adaptive Vocabulary Construction for Frustration Intensity Modelling in Customer Support Dialog Texts

This paper examines emotion intensity prediction in dialogs between clients and customer support representatives occurring on Twitter. We focus on a single emotion type, namely, frustration, modelling the user's level of frustration while attempting to predict frustration intensity on the current and next turn, based on the text of turns coming from both dialog participants. A subset of the Kaggle Customer Support on Twitter dataset was used as the modelling data, annotated with per-turn frustration intensity ratings. We propose to represent dialog turns by binary encoded bags of automatically selected keywords to be subsequently used in a machine learning classifier. To assess the classification quality, we examined two different levels of accuracy imprecision tolerance. Our model achieved a level of accuracy significantly higher than a statistical baseline for prediction of frustration intensity for a current turn. However, we did not find the additional information from customer support turns to help predict frustration intensity of the next turn, and the reason for that is possibly the stability of user’s frustration level over the course of the conversation, in other words, the inability of support’s response to exert much influence to user’s initial frustration level.