The Ebb and Flow of Deep Learning: a Theory of Local Learning.

摘要: In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting synaptic weights can only depend on variables that available locally, such as activity of pre- post-synaptic neurons, resulting in local learning rules. A systematic framework studying space must first define nature variables, then functional form ties them together into each rule. We consider polynomial analyze their behavior capabilities both linear non-linear networks. As byproduct, this enables also discovery new well important relationships between group symmetries. Stacking deep feedforward networks leads to learning. While learn interesting representations, it cannot complex input-output functions, even when targets top layer. Learning functions requires target information is propagated layers through backward channel. The about targets, channel which propagated, partition algorithms. For any algorithm, capacity be defined number bits provided gradient per weight, divided by required operations weight. estimate associated with several algorithms show backpropagation outperforms achieves maximum possible capacity. theory clarifies concept Hebbian learning, what learnable explains sparsity discovered so far.