Sparse hidden units activation in Restricted Boltzmann Machine
作者： Jakub M. Tomczak , Adam Gonczarek
DOI: 10.1007/978-3-319-08422-0_27
关键词:
摘要: Sparsity has become a concept of interest in machine learning for many years. In deep sparse solutions play crucial role obtaining robust and discriminative features. this paper, we study new regularization term hidden units activation the context Restricted Boltzmann Machine (RBM). Our proposition is based on symmetric Kullback-Leibler divergence applied to compare actual desired distribution over active units. We our method against two other enforcing sparsity terms by evaluating empirical classification error using datasets: (i) image (MNIST), (ii) document (20-newsgroups).
springer.com
sci-hub.st 参考文章(22)
Geoffrey E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines Neural Networks: Tricks of the Trade (2nd ed.). pp. 599- 619 ,(2012) , 10.1007/978-3-642-35289-8_32
KyungHyun Cho, Alexander Ilin, Tapani Raiko, Tikhonov-Type Regularization for Restricted Boltzmann Machines Artificial Neural Networks and Machine Learning – ICANN 2012. pp. 81- 88 ,(2012) , 10.1007/978-3-642-33269-2_11
David E. Rumelhart, James L. McClelland, , Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations Computational Models of Cognition and Perception. ,(1986) , 10.7551/MITPRESS/5236.001.0001
Christopher M. Bishop, Pattern Recognition and Machine Learning ,(2006)
Geoffrey E. Hinton, Vinod Nair, Rectified Linear Units Improve Restricted Boltzmann Machines international conference on machine learning. pp. 807- 814 ,(2010)
P. Smolensky, Information processing in dynamical systems: foundations of harmony theory Parallel distributed processing: explorations in the microstructure of cognition, vol. 1. pp. 194- 281 ,(1986)
Ilya Sutskever, Geoffrey E. Hinton, Alex Krizhevsky, Ruslan R. Salakhutdinov, Nitish Srivastava, Improving neural networks by preventing co-adaptation of feature detectors arXiv: Neural and Evolutionary Computing. ,(2012)
Hugo Larochelle, Yoshua Bengio, Classification using discriminative restricted Boltzmann machines Proceedings of the 25th international conference on Machine learning - ICML '08. pp. 536- 543 ,(2008) , 10.1145/1390156.1390224
Aaron Courville, Guillaume Desjardins, James Bergstra, Yoshua Bengio, The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions. IEEE Transactions on Pattern Analysis and Machine Intelligence. ,vol. 36, pp. 1874- 1887 ,(2014) , 10.1109/TPAMI.2013.238
Nicolas Le Roux, Yoshua Bengio, Representational power of restricted boltzmann machines and deep belief networks Neural Computation. ,vol. 20, pp. 1631- 1649 ,(2008) , 10.1162/NECO.2008.04-07-510