Learned-norm pooling for deep feedforward and recurrent neural networks

摘要: In this paper we propose and investigate a novel nonlinear unit, called L p for deep neural networks. The proposed unit receives signals from several projections of subset units in the layer below computes normalized norm. We notice two interesting interpretations unit. First, can be understood as generalization number conventional pooling operators such average, root-mean-square max widely used in, instance, convolutional networks (CNN), HMAX models neocognitrons. Furthermore, is, to certain degree, similar recently maxout [13] which achieved state-of-the-art object recognition results on benchmark datasets. Secondly, provide geometrical interpretation activation function based argue that is more efficient at representing complex, separating boundaries. Each defines superelliptic boundary, with its exact shape defined by order p. claim makes it possible model arbitrarily shaped, curved boundaries efficiently combining few different orders. This insight justifies need learning orders each model. empirically evaluate datasets show multilayer perceptrons (MLP) consisting achieve recurrent (RNN).