Learning to recognise behaviours of persons with dementia using multiple cues in an HMM-based approach

This paper presents a learning technique for visual event recognition in a system that assists persons with dementia during handwashing. The challenge is that persons with dementia present a wide variety of behaviors during a single task, typically changing their behaviours drastically from day to day. Any attempt at modeling this variety requires a large set of features, image regions, and temporal dynamics. In this paper, we approach this challenge by supervised learning of generative models from manually segmented and labelled video sequences. Our method uses a generic set of appearance-based colour, motion and texture features, over a static set of regions. We then present two HMM architectures that incorporate multiple image regions by either fusing on a feature-level, or later in the recognition process using a mixture-of-experts approach, in which a gating HMM is applied for the dynamic selection between specialised expert HMMs. Our models are trained on a clinical database of videos, and we compare the HMM approaches with a nearest neighbours scheme. Our results confirm the challenge we present, and indicate that our generative modelling techniques are suitable for inclusion in future prototypes of the hand washing assistant.