Semantic Matching Evaluation of User Responses to Electronics Questions in AutoTutor

摘要: Relatedness between user input and an ideal response is a salient feature required for proper functioning of Intelligent Tutoring System (ITS) using natural language processing. Improper assessment text causes maladaptation in ITSs. Meta-assessment responses ITSs can improve instruction efficacy satisfaction. Therefore, this paper evaluates the quality semantic matching expected AutoTutor, ITS which holds conversation with language. AutoTutor's dialogue driven by AutoTutor Conversation Engine (ACE), uses combination Latent Semantic Analysis (LSA) Regular Expressions (RegEx) to assess input. We assessed ACE via from 219 Amazon Mechanical Turk users, who answered 118 electronics questions broken into 5202 pairings (n = 5202). These analyses explore relationship RegEx LSA, agreement two judges, human judges ACE. Additionally, we calculated precision recall. As expected, regular expressions LSA had moderate, positive relationship, was fair, but slightly lower than human.