Knowledge distillation for mixture of experts models in speech recognition

摘要: The sparsely-gated mixture of experts (MoE) architecture can scale out large Transformer models to orders of magnitude which are not achievable by dense models with the current …