The Shape of the Moving Image

Abstract

Driven by cognitive studies of visuo-auditory narrative media, our research focusses on developing methods for the machine coding of the shape of the moving image from the viewpoints of artificial intelligence, visuo-spatial cognition, and computational models of narrative. Our interpretation of the shape of the moving image encompasses two key aspects:

— the geometry of a scene, based on its visuo-spatial cinematographic scene and narrative structure and corresponding underlying embodied image-schematic semantics — visual reception of the moving image by observers (analysed using eye-tracking methods)

This talk presents a high-level summary of our machine coding method: a cognitive-computational framework for the machine-based grounding and semantic interpretation of "dynamic visuo-spatial imagery" consisting of video and eye-tracking data. We demonstrate key technological capabilities aimed at investigating attention and recipient effects vis-a-vis the motion picture; this encompasses high-level analysis of subject's visual fixation patterns and correlating this with “deep semantic analysis" of the visuo-auditory data (e.g., fixation on movie characters, influence of cinematographic devices such as ‘cuts' and 'sound effects' on attention). As case-study, select examples are presented from a large-scale eye-tracking experiment. The talk will emphasis the machine coding framework and its application as a general cognitive assistive technology platform for visual perception research in general, and cognitive media studies in particular.