This master project is part of the Social Media Analytics course carried out in the Swiss Joint Master in Computer Science.

This paper was written for the Master seminar of Human Computer Interaction proposed by the Human-IST Institute, and supervised by the Dr. Simon Ruffieux, member of the research group.

With the advent of autonomous cars, mobile devices and conversational agents, the question of the interaction with digital devices in everyday life is becoming more and more relevant. The aim of this seminar was to look at this question over several specific contexts and expose students to state-of-the art research in Human-Computer Interaction.

In my research work, I addressed the challenge of evaluating the performance of Conversational Agents (CA) in the domain of Human Machine Teaming systems. Firstly, I discussed the current evaluation methods and exposed the lack of a defined procedure to assess the performance of a CA.

To overcome this challenge, I proposed the performance metric P, which incorporates general and domain-specific measures to provide a complete assessment of a CA. In addition, I proposed a visual representation to aid in the interpretation of the results.

To validate the usefulness of P as a comprehensive HMT metric for assessing the performance of the machine, the human, and their interaction, I outlined an experiment on two different CAs that demonstrates its assessment capabilities.