IIITB researchers develop pipeline for a ‘human’ engagement with bots

Along with the prediction of engagement, the pipeline can provide deeper insights into human personality, attitudes, and emotions and thereby improve the way the robot interacts with the human user.
| Photo Credit: Illustration: Sebastian Francis

While a teacher can always infer if a student is paying attention in the classroom, would an edutech bot know if the lesson on screen is holding his/her attention? The day may not be too far when the distinguishing line between these two kinds of interactions would blur.

Researchers at the International Institute of Informational Technology Bangalore (IIITB) have developed a ready-to-use interpretable pipeline which uses machine learning and psychology for modelling engagement in human-robot interactions. Along with the prediction of engagement, the pipeline can provide deeper insights into human personality, attitudes, and emotions and thereby improve the way the robot interacts with the human user. 

Sense of human emotions

“The robotic interaction systems we have now do not take into account the human feelings, emotions and attitudes at a particular moment whereas in human interactions, we take these factors into account. We want assistive robots, chatbots, and other automated systems to also have a sense of human emotions,” Shrisha Rao, professor at IIIT Bangalore, and senior researcher on the study, told The Hindu. 

While interactive robots are expected to adapt to each individual’s needs when dealing with diverse engagement levels among various people in a crowd, the accurate measurement of each person’s engagement based on their visual and physiological cues is the initial challenge. This is where the pipelines that can deduce these engagement levels come into picture.  

The study which has been published in the PLOS One Journal of the Public Library Of Science has been authored by Soham Joshi and Arpitha Malavalli, graduate students in IIITB, under the supervision of Mr. Rao.  

The Big Five personality traits (extroversion, agreeableness, openness, conscientiousness, and neuroticism), the Interpersonal Circumplex (IPC), and the Triandis Theory of Interpersonal Behavior have been incorporated into the pipeline for the prediction of engagement. 

“Many previous methods for predicting engaging behaviour rely on blindly training deep neural networks to understand the link between multimodal cues and engagement levels. However, this approach differs significantly from how humans assess engagement in their interaction. Our goal is to create a prediction pipeline in human-robot interactions that mimic this human-like assessment of engagement,” said Mr. Joshi. 

Explaining the various stages of the pipeline, Mr. Joshi said, “We will use incoming multimodal cues like head movements, hand gestures, eye movements, and physiological features to predict the Big Five Personality traits of the interacting human. The personality predictions are then projected on IPC, and the correlation values between human behaviour and attitude and emotion are used to get inferences about the engaging behaviour displayed by a human interacting with a robot.”

Wide range of applications

This pipeline can have a wide range of applications in various domains such as online learning platforms, assistive robotics, and intelligent conversational agents. While the hardware required for the application could vary across domains, going forward the researchers would like to partner with domains to see if products or solutions can be developed with the pipeline. They would also like to focus on shrinking the bulky data set that is required in the back end.

Elaborating on the applications, Mr. Rao said, “For instance, the inference of lower engagement in customers using chat-bots may indicate a need for a change in the chat-bots interacting style to cater to humans of different attitudes and emotions. In online learning, the pipeline can be used to predict low engagement levels among students and introduce gamification as a way to incentivise them.”