Conference Paper Affect-as-Information Approach to a Sentiment Analysis Based Evaluation of Conversational Agents

Ptaszynski, Michal  ,  Dybala, Pawel  ,  Higuchi, Shinsuke  ,  Rzepka, Rafal  ,  Araki, Kenji

In this paper we propose a novel method forautomatic evaluation of conversational agents. Themethod is based on analyzing the user’s affectconveyed in utterances. From analyzing: the user’sgeneral emotional engagement in the conversation andthe emotion types conveyed by the user in theconversation, a simple psychological reasoning isderived about the user’s sentiment about the agent’sperformance. The evaluation experiment on twoJapanese-speaking conversational agents showed thesame tendencies in the results returned by the systemconstructed on the proposed method and the user’sopinion about the two agents checked in the afterwardsurvey. Thus the method can be used for evaluation ofJapanese-speaking conversational agents.

Number of accesses :  

Other information