AIRCC's International Journal of Computer Science and Information Technology

Open Access Open Access  Restricted Access Subscription Access

Adaptive Vocabulary Construction for Frustration Intensity Modelling in Customer Support Dialog Texts


Affiliations
1 Department of Computer Science, University of Latvia, Riga, Latvia
 

This paper examines emotion intensity prediction in dialogs between clients and customer support representatives occurring on Twitter. We focus on a single emotion type, namely, frustration, modelling the user's level of frustration while attempting to predict frustration intensity on the current and next turn, based on the text of turns coming from both dialog participants. A subset of the Kaggle Customer Support on Twitter dataset was used as the modelling data, annotated with per-turn frustration intensity ratings. We propose to represent dialog turns by binary encoded bags of automatically selected keywords to be subsequently used in a machine learning classifier. To assess the classification quality, we examined two different levels of accuracy imprecision tolerance. Our model achieved a level of accuracy significantly higher than a statistical baseline for prediction of frustration intensity for a current turn. However, we did not find the additional information from customer support turns to help predict frustration intensity of the next turn, and the reason for that is possibly the stability of user’s frustration level over the course of the conversation, in other words, the inability of support’s response to exert much influence to user’s initial frustration level.

Keywords

Neural Networks, Emotion Annotation, Emotion Recognition, Emotion Intensity, Frustration.
User
Notifications
Font Size

  • P. Ekman, (1992) “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3-4, pp.169–200.

  • C. O. Alm, D. Roth, and R. Sproat, “Emotions from text,” Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT 05, 2005, pp.579–586.

  • A. Balahur, J. M. Hermida, A. Montoyo, and R. Muñoz, (2013) “Detecting implicit expressions of affect in text using EmotiNet and its extensions,” Data & Knowledge Engineering, Vol. 88, pp.113–125.

  • E. C.-C. Kao, C.-C. Liu, T.-H. Yang, C.-T. Hsieh, and V.-W. Soo, (2009) “Towards Text-based Emotion Detection A Survey and Possible Improvements,” 2009 International Conference on Information Management and Engineering, pp.70-74.

  • A. Al-Mahdawi and W.J. Teahan (2019) “Automatic emotion recognition in English and Arabic text” (Doctoral dissertation, Bangor University).

  • S. Lee and Z. Wang, (2015) “Emotion in Code-switching Texts: Corpus Construction and Analysis,” Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pp.91-99.

  • V. Duppada and S. Hiray, (2017) “Seernet at EmoInt-2017: Tweet Emotion Intensity Estimator,” Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp.205-211.

  • R. González-Ibánez, S. Muresan, and N. Wacholder, (2011). Identifying sarcasm in Twitter: a closer look. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.581-586.

  • G. Badaro, H. Jundi, H. Hajj, and W. El-Hajj, (2018) “EmoWordNet: Automatic Expansion of Emotion Lexicon Using English WordNet,” Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp.86-93.

  • A. Reyes, P. Rosso, and T. Veale, (2012) “A multidimensional approach for detecting irony in Twitter,” Language Resources and Evaluation, Vol. 47, No. 1, pp. 239–268.

  • J. Klein, Y. Moon and R.W. Picard, R.W., (2002). “This computer responds to user frustration: Theory, design, and results.” Interacting with computers, Vol. 14, No. 2, pp.119-140.

  • K. Hone, (2006). “Empathic agents to reduce user frustration: The effects of varying agent characteristics.” Interacting with computers, Vol. 18, No. 2, pp.227-245.

  • T. Hu, A. Xu, Z. Liu, Q. You, Y. Guo, V. Sinha, J. Luo, and R. Akkiraju, (2018) “Touch Your Heart,” Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI 18, pp.1-12.

  • P. Goel, D. Kulshreshtha, P. Jain and K.K. Shukla, (2017) Prayas at emoint 2017: An ensemble of deep neural architectures for emotion intensity prediction in tweets. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 58-65.

  • F. Bravo-Marquez, E. Frank, B. Pfahringer, and S. M. Mohammad, (2019). Affective tweets: A weka package for analyzing affect in tweets. Journal of Machine Learning Research, Vol. 20, No. 92, pp.1–6.

  • B. Byrne, K. Krishnamoorthi, C. Sankar, A. Neelakantan, B. Goodrich, D. Duckworth, S. Yavuz, A. Dubey, K.-Y. Kim, and A. Cedilnik, (2019) “Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

  • Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, (2017) Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.

  • D. Ham, J. G. Lee, Y. Jang, and K. E. Kim, (2020) End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.583–592.

  • P. Colombo, W. Witon, A. Modi, J. Kennedy, and M. Kapadia, (2019) “Affect-Driven Dialog Generation,” Proceedings of the 2019 Conference of the North, pp. 3734–3743

  • N. Lubis, S. Sakti, K. Yoshino, and S. Nakamura, (2018). Eliciting positive emotion through affect-sensitive dialogue response generation: A neural network approach. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp.5293–5300.

  • A. Xu, Z. Liu, Y. Guo, V. Sinha, and R. Akkiraju, (2017) “A New Chatbot for Customer Service on Social Media, ” Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3506–3510.

  • Ļeonova V., (2020) "Review of Non-English Corpora Annotated for Emotion Classification in Text." International Baltic Conference on Databases and Information Systems. pp. 96-108.


Abstract Views: 362

PDF Views: 131




  • Adaptive Vocabulary Construction for Frustration Intensity Modelling in Customer Support Dialog Texts

Abstract Views: 362  |  PDF Views: 131

Authors

Janis Zuters
Department of Computer Science, University of Latvia, Riga, Latvia
Viktorija Leonova
Department of Computer Science, University of Latvia, Riga, Latvia

Abstract


This paper examines emotion intensity prediction in dialogs between clients and customer support representatives occurring on Twitter. We focus on a single emotion type, namely, frustration, modelling the user's level of frustration while attempting to predict frustration intensity on the current and next turn, based on the text of turns coming from both dialog participants. A subset of the Kaggle Customer Support on Twitter dataset was used as the modelling data, annotated with per-turn frustration intensity ratings. We propose to represent dialog turns by binary encoded bags of automatically selected keywords to be subsequently used in a machine learning classifier. To assess the classification quality, we examined two different levels of accuracy imprecision tolerance. Our model achieved a level of accuracy significantly higher than a statistical baseline for prediction of frustration intensity for a current turn. However, we did not find the additional information from customer support turns to help predict frustration intensity of the next turn, and the reason for that is possibly the stability of user’s frustration level over the course of the conversation, in other words, the inability of support’s response to exert much influence to user’s initial frustration level.

Keywords


Neural Networks, Emotion Annotation, Emotion Recognition, Emotion Intensity, Frustration.

References