Audio-visual speech translation with automatic lip syncqronization and face tracking based on 3-D head model

摘要: Speech-to-speech translation has been studied to realize natural human communication beyond language barriers. Toward further multi-modal communication, visual information such as face and lip movements will be necessary. In this paper, we introduce a English-to-Japanese Japanese-to-English system that also translates the speaker's speech motion while synchronizing it translated speech. To retain facial expression, substitute only organ's image with synthesized one, which is made by three-dimensional wire-frame model adaptable any speaker. Our approach enables synthesis an extremely small database. We conduct subjective evaluation connected digit discrimination using data without audiovisual lip-synchronicity. The results confirm sufficient quality of proposed audio-visual system.