Comparative Evaluation and Shared Tasks for NLG in Interactive Systems

摘要: Introduction Natural Language Generation (NLG) has strong evaluation traditions, in particular the area of user NLG-based application systems, as conducted for example M-PIRO (Isard et al ., 2003), COMIC (Foster and White, 2005), SumTime (Reiter Belz, 2009) projects. There are also examples embedded NLG components compared to non-NLG baselines, including, e.g., DIAG (Di Eugenio 2002), STOP 2003b), SkillSum (Williams Reiter, 2008) evaluations, different versions same component, ILEX (Cox 1999), SPoT (Rambow 2001), CLASSiC (Janarthanam 2011) Starting with Langkilde Knight's work (Knight Langkilde, 2000), automatic against reference texts began be used, especially surface realization. What was missing, until 2006, were comparative results directly comparable, but independently developed, systems. In 1981, Sparck Jones wrote that information retrieval (IR) lacked consolidation ability progress collectively, this substantially because there no commonly agreed framework describing evaluating systems (Sparck Jones, p. 245). Since then, various sub-disciplines natural language processing (NLP) speech technology have consolidated progressed collectively through developing common task definitions frameworks, context shared-task campaigns (STECs), achieved successful commercial deployment a range technologies (e.g. recognition software, document retrieval, dialogue systems).