The impact of evaluation scenario development on the quantitative performance of speech translation systems prescribed by the SCORE framework

摘要: The Defense Advanced Research Projects Agency's (DARPA) Spoken Language Communication and Translation for Tactical Use (TRANSTAC) program is a focused advanced technology research development program. intent of this to demonstrate capabilities quickly develop implement free-form, two-way, speech-to-speech spoken language translation systems allowing speakers different languages communicate with each other in real-world tactical situations without the need an interpreter. National Institute Standards Technology (NIST), support from Mitre Corporation Appen Pty Limited, has been funded by DARPA evaluate TRANSTAC technologies since 2006. NIST-led Independent Evaluation Team (IET) numerous responsibilities ongoing effort including collecting processing training data, designing implementing performance evaluations, analyzing test data. In order design execute fair relevant NIST IET employed System, Component Operationally-Relevant (SCORE) framework. SCORE framework unified set criteria tools built around premise that, gain understanding how would perform its intended environment, it must be evaluated at both component system levels further tested operationally-relevant environments while capturing quantitative qualitative Since evaluation goal capture data technologies, developed implemented SCORE-inspired live scenarios. two forms scenarios have unique impacts on This paper presents methodology, as well their influence performance.