Multimodal feedback loop design for sustained attention

Multimodal feedback loop design for sustained attention

This project looks at diverse computer vision-based indicators of sustained attention in computer-mediated learning scenarios. To form a multimodal feedback loop for better-sustained attention, multimodal machine learning is used for the learning analytics, while multimodal feedback via conversational agents provides timely learning supports.

About this project

In this project, we look at diverse computer vision-based indicators of sustained attention in computer-mediated learning scenarios. The sustained attention level of learners is monitored during learning activities, while diverse feedback with multimodality enhances learning outcomes and the perceived learning experiences, forming a multimodal feedback loop for better-sustained attention.

Research Approach

We focus on multimodality as a means to understand learner status and learning environments. We mainly use video modality for computer vision-based machine learning and mix multimodal indicators to find cues for attention fluctuation. On a feedback provision level, we involve multiple modalities via various agents, including an interface based on a computer as well as conversational agents, mainly humanoid robots. We then look at which interface, modality combinations, and feedback feature better support attention management during computer-mediated learning.

Expected outcomes

We have two objectives in this project: 1) multimodal sustained attention detection and 2) feedback generation for better attention management. Attention is often considered as something we cannot capture well or manage due to its various interpretations and its difficult timely feedback provision. Through the first part of this project, we aim to successfully detect attention levels by finding the ground truth of sustained attention using a multimodal machine learning approach, using various psychological and statistical analysis methods. We connect our findings from the first project with the second phase of the project, which is the feedback provision. We aim to provide feedback with various modalities which suit learning activities and environments the best for learners on subjective and objective levels.

Partners

Faculty Industrial Design Engineering - TU Delft
Faculty Electrical Engineering, Mathematics and Computer Science - TU Delft