Some multimodal signals in humans

摘要: In this paper, I will give an overview of some well-studied multimodal signals that humans produce while they communicate with other humans, and discuss the implications those studies for HCI. first a conceptual framework allows us to distinguish between functional sensory modalities. This distinction is important, as there are multiple modalities using same modality (e.g., facial expression eye-gaze in visual modality). A second theoretically important issue redundancy. Some appear be redundant signal another modality, whereas others new information or even conflicting (see e.g., work Susan Goldin-Meadows on speech accompanying gestures). argue never truly redundant. First, many gestures at sight express meaning generally provide extra (analog) about manner, path, etc. Second, simple fact expressed more than one itself communicative signal. Armed background, then proceed have been investigated human-human research, level understanding we signals. The latter especially potential implementations these artificial agents. pointing gestures. address timing relative it supposed support, mutual dependency speech, existence alternative ways from cultures. most frequent form does not involve index finger cultural practice called lip-pointing which employs two modalities, mouth-shape eye-gaze, simultaneously pointing. Next, eye-gaze. classical study by Kendon (1967) claims systematic relationship (at interlocutor) turn-taking states. Research our institute has shown weaker often assumed. If dialogue setting contains visible object relevant map), rate eye-gaze-at-other drops dramatically its turn taking disappears completely. machine generated discussed. Finally, explore theoretical debate regarding spontaneous It claimed class iconic McNeill (1992) “window into mind”. That is, researcher (or direct view speaker’s thought, without being obscured complex transformation take place when transforming thought verbal utterance. illusion. Gestures can specifically designed such listener expected interpret them. Although transformations carried out gesture indeed (partly) different corresponding a) complex, b) severely understudied. obviously consequences both research agenda, generation machines.