Learning deep hierarchical and temporal recurrent neural networks with residual learning

摘要: Learning both hierarchical and temporal dependencies can be crucial for recurrent neural networks (RNNs) to deeply understand sequences. To this end, a unified RNN framework is required that ease the learning of deep structures by allowing gradients propagate back from ends without being vanished. The residual (RL) has appeared as an effective less-costly method facilitate backward propagation gradients. significance RL exclusively shown representations dependencies. Nevertheless, there lack efforts unify these finding into single RNNs. In study, we aim prove approximating identity mapping optimizing structures. We propose called RNNs, learn RNNs mappings across validate proposed method, explore efficacy employing shortcut connections training sequence problems. Experiments are performed on Penn Treebank, Hutter Prize IAM-OnDB datasets results demonstrate utility in terms accuracy computational complexity. even large exploiting parameters increasing network depth gain benefits with reduced size "state".