Multilingual Probing of Deep Pre-Trained Contextual Encoders

摘要: Encoders that generate representations based on context have, in recent years, benefited from adaptations allow for pre-training large text corpora. Earlier work evaluating fixed-length sentence has included the use of ‘probing’ tasks, diagnostic classifiers to attempt quantify extent which these encoders capture specific linguistic phenomena. The principle probing also resulted extended evaluations include relatively newer word-level pre-trained encoders. We build tasks established literature and comprehensively evaluate analyse – a typological perspective amongst others multilingual variants existing datasets constructed 6 non-English languages. Specifically, we probe each layer multiple monolingual RNN-based ELMo models, transformer-based BERT’s cased uncased variants, variant BERT uses cross-lingual modelling scheme (XLM).