Alignment of Speech and Co-speech Gesture in a Constraint-based Grammar

摘要: This thesis concerns the form-meaning mapping of multimodal communicative actions consisting speech signals and improvised co-speech gestures, produced spontaneously with hand. The interaction between speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish underlying mechanisms for synchronous gesture production, also computational build computer systems that communicate through multiple modalities. Based on findings this previous research, we advance new theory in which form combined speech-and-gesture signal its meaning is analysed constraint-based grammar. We propose several construction rules about well-formedness motivate empirically an extensive detailed corpus study. In particular, use prosody, syntax semantics speech, signal, as well temporal performance relative constrain derivation single tree turn determines representation via standard semantic composition. Gestural often underspecifies meaning, so output our grammar underspecified logical formulae support range possible interpretations act final context-of-use, given current models semantics/pragmatics interface. It held community co-expressivity determined basis their co-occurrence: is, semantically related happened at same time gesture. Whereas usually taken granted, methodology establishing systematic domain-independent way spoken element(s) can be to, based form, yield supports intended interpretation(s) context. ‘semantic’ alignment thus driven not co-occurrence alone, but linguistic properties overlaps with. doing, contribute fine-grained system articulating uses methods linguistics. show just language exhibits ambiguity both do actions: instance, integration restricted unique phrase rather aligned iii trees yielding distinct representations. These mappings stem fact derived highly incomplete even An overall challenge account action context using linguistics syntactic