Bayesian Compression for Deep Learning
作者： Max Welling , Karen Ullrich , Christos Louizos
DOI:
关键词:
摘要: Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled effective way to attack is by adopting Bayesian point view, where through sparsity inducing priors prune large parts network. We introduce two novelties paper: 1) use hierarchical nodes instead individual weights, 2) posterior uncertainties determine optimal fixed precision encode weights. Both factors significantly contribute achieving state art terms compression rates, while still staying competitive with methods designed optimize for speed or energy efficiency.
arxiv.org参考文章(62)
D. F. Andrews, C. L. Mallows, Scale Mixtures of Normal Distributions Journal of the Royal Statistical Society: Series B (Methodological). ,vol. 36, pp. 99- 102 ,(1974) , 10.1111/J.2517-6161.1974.TB00989.X
Neil D. Lawrence, Note Relevance Determination Perspectives in Neural Computing. pp. 128- 133 ,(2002) , 10.1007/978-1-4471-0219-9_11
Eric Nalisnick, Anima Anandkumar, Padhraic Smyth, A Scale Mixture Perspective of Multiplicative Noise in Neural Networks arXiv: Machine Learning. ,(2017)
Geoffrey Hinton, Radford M. Neal, Bayesian learning for neural networks ,(1995)
Gunnar Rätsch, Theofanis Karaletsos, Automatic Relevance Determination For Deep Generative Models arXiv: Machine Learning. ,(2015)
Kilian Q. Weinberger, Wenlin Chen, Yixin Chen, James T. Wilson, Stephen Tyree, Compressing Convolutional Neural Networks. arXiv: Learning. ,(2015)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification international conference on computer vision. pp. 1026- 1034 ,(2015) , 10.1109/ICCV.2015.123
Yunchao Gong, Lubomir D. Bourdev, Liu Liu, Ming Yang, Compressing Deep Convolutional Networks using Vector Quantization arXiv: Computer Vision and Pattern Recognition. ,(2014)
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, Distilling the Knowledge in a Neural Network arXiv: Machine Learning. ,(2015)
Ankur Agrawal, Pritish Narayanan, Kailash Gopalakrishnan, Suyog Gupta, Deep Learning with Limited Numerical Precision arXiv: Learning. ,(2015)