Individual and team profiling to support theory of mind in artificial social intelligence

Abstract

We describe an approach aimed at helping artificial intelligence develop theory of mind of their human teammates to support team interactions. We show how this can be supported through the provision of quantifiable, machine-readable, a priori information about the human team members to an agent. We first show how our profiling approach can capture individual team member characteristic profiles that can be constructed from sparse data and provided to agents to support the development of artificial theory of mind. We then show how it captures features of team composition that may influence team performance. We document this through an experiment examining factors influencing the performance of ad-hoc teams executing a complex team coordination task when paired with an artificial social intelligence (ASI) teammate. We report the relationship between the individual and team characteristics and measures related to task performance and self-reported perceptions of the ASI. The results show that individual and emergent team profiles were able to characterize features of the team that predicted behavior and explain differences in perceptions of ASI. Further, the features of these profiles may interact differently when teams work with human versus ASI advisors. Most strikingly, our analyses showed that ASI advisors had a strong positive impact on low potential teams such that they improved the performance of those teams across mission outcome measures. We discuss these findings in the context of developing intelligent technologies capable of social cognition and engage in collaborative behaviors that improve team effectiveness.

Figure 1

The two dimensions of our profiling approach were Taskwork Potential and Teamwork Potential each defined by scores on three measures. Taskwork potential was determined by: spatial ability (SSOD: Santa Barbara Sense of Direction), video gaming experience (VGE: Video Gaming Experience), and a task skills competency test (COMP: Competency test). Teamwork potential was determined by: preferences for team and group work (Collectivism: Psychological Collectivism), attitudes towards social interactions (SD: Sociable Dominance), and social intelligence (RMET: Reading the Mind in the Eyes Test). Team members were categorized as either high or low in potential on each dimension. The four cells in this graph demonstrate the most extreme possible categorizations with team members in the upper left quadrant demonstrating low potential on both dimensions, and members in the lower right demonstrating high potential on both.

Figure 2

Team members were categorized on Taskwork Potential and Teamwork potential based on their scores relative to the median of the population scores on each measure comprising each dimension. To be categorized as high potential in with Taskwork or Teamwork, participants had to score above the median on at least two of the three measures for the respective dimension. The top two radar charts demonstrate different distributions of high/low scores a participant might produce while still being categorized as high on both dimensions. The bottom two radar charts demonstrate different scores that could possibly result in being categorized as low in Teamwork Potential, but high in Taskwork potential. Note that this is not an exhaustive listing, but rather examples of possible scoring combinations.

Figure 3

Results showed that there was an interaction between individuals’ teamwork potential profile and taskwork potential profile such that high teamwork but low taskwork potential was associated with the highest ratings of ASI advisors while low teamwork and low taskwork potential was associated with the lowest ratings. Rather, individuals that did not exhibit the skills to succeed on their own but were geared towards teamwork seemed to value the ASI advisors more highly than those teammates that had high taskwork potential.

Figure 4

The interaction between teams’ teamwork and taskwork potential profiles was found to account for a moderate proportion of variance in taskwork-teamwork measure 1 (mission score %). Notably, teams that were high in taskwork potential and high in teamwork potential attained higher mission scores, in terms of percent of the maximum possible score.

Figure 5

Mission score by advisor condition. Here, the impact of ASI advisors on low taskwork, low teamwork potential teams can be observed clearly. Interestingly, ASI may have also reduced the performance of high taskwork, high teamwork potential teams, highlighting the importance of appropriate assignment and employment of artificially intelligent assistants.

Figure 6

The interaction between teams’ holistic profiles and the nature of their mission advisor accounted for a large proportion of variance in teamwork measure 1 (knowledge externalization behaviors). Most importantly, it can again be observed in this visualization that ASI advisors had a notable positive effect on the behaviors of low taskwork, low teamwork potential teams such that they improved not only their task execution but also their engagement in teamwork actions. Conversely, teams that were low in taskwork and teamwork potential and had no advisor failed to employ knowledge externalization tools effectively. Teams working with a human or ASI advisor tended to employ externalization tools more efficiently to share information and achieve coordination. Those teams that were high in taskwork and teamwork potential but had no advisor tended to overuse externalization tools.