According to the latest research results of Intel China Research Institute, this article mainly introduces a dynamic surgery algorithm that effectively solves the problem of long retraining time and high rate of false pruning when dealing with large networks. Using this algorithm, we can easily compress the total number of parameters of the two classic networks, LeNet and AlexNet, by a factor of 108 and 17.7, respectively.
Intel China Research Institute recently proposed a neural network compression algorithm called “dynamic surgery†that has received extensive attention from the industry and high appraisal from domestic and foreign experts. Using this method, researchers can greatly reduce the network structure without sacrificing the performance of the original model. Let us take you to understand this algorithm in a simple and easy way.
If you have a habit of paying regular attention to IT and Internet news, you must not feel unfamiliar with deep learning. The concept of deep learning stems from the early research on artificial neural networks. The core of its concept is to complete the “learning†of abstract representation of sample data through deep neural networks . Since the concept of deep learning was first proposed in 2006, it has greatly changed the ecology of artificial intelligence and even the entire Internet and IT fields. After a decade of development, the technology has shown top performance in many areas including face recognition, speech recognition, object detection, and natural language processing.
One of the mainstream deep networks: deep convolutional neural networks
(Source: A tutorial on deep learning [3])
However, there is still a certain distance between deep learning and “invincibilityâ€. One of the major bottlenecks restricting its development is the conflict between the extremely high model complexity of deep networks and the extremely limited hardware resources of consumer electronics. In fact, many mainstream deep networks nowadays contain learning parameters of tens of millions or even billions of dollars, and such a large number of parameters bring considerable pressure to the storage and calculation of models. Therefore, how to compress a well-trained deep network has become a major problem for researchers. The 2015 paper, Learning both weights and connections for efficient neural networks, proposes a network pruning algorithm that can compress learning parameters in deep networks more than 10 times on the premise of maintaining data representation capabilities. Caused extensive discussion. The paper was also published in the Conference on Neural Information Processing Systems (NIPS) at the top of the field of machine learning and has gained tremendous influence.
Neural network pruning strategy
(Source: Learning both weights and connections for efficient neural networks [1])
This year, the three researchers of Guo Yiwen, Yao Anbang, and Chen Yurong, Intel Cognitive Computing Lab of Intel China, made breakthrough progress in deep learning. The dynamic network surgery algorithm proposed by them effectively solves the problem of long retraining time and high rate of false pruning when dealing with large networks. Using this algorithm, we can easily compress the total number of parameters of the two classic networks, LeNet and AlexNet, by a factor of 108 and 17.7, respectively.
Intel China Institute of Cognitive Computing Lab Guo Yiwen, Chen Yurong and Yao Anbang
The algorithm adopts the strategy of combining pruning and grafting, training and compression to complete the network compression task. Through the introduction of the network grafting operation, the performance loss caused by wrong pruning is avoided, so that the theoretical limit of network compression is better approximated in practical operations.
Surgical strategy
(The dashed line represents the currently cut off network connection, while the green line represents the reestablished network connection through grafting [2])
At present, this method has been written and will be published at this year's NIPS conference. As the top conference in the field of machine learning, NIPS has maintained a very low acceptance rate of papers in recent years. The papers that are approved for publication will also receive great attention from people in the industry, which will give a considerable impetus to the development of machine learning. It is reported that the total number of manuscripts received by NIPS this year is about 2,500, and the acceptance rate is less than 23%. I believe that this work of Intel China Research Institute will have a tremendous impact on the academic and industrial sectors.
Image Source:
[1] Han, Song, et al. "Learning both weights and connections for efficient neural networks." Advances in Neural Information Processing Systems, 2015.
[2] Guo, Yiwen, Yao, Anbang and Chen, Yurong. “Dynamic network surgery for efficient DNNs.†Arixv preprint, arXiv: 1608.04493v1, 2016.
[3] Yu, Kai. “A tutorial on deep learning.†China Workshop on Machine Learning and Applications, 2012.
Lei Feng network (search "Lei Feng network" public number attention) Note: This article is authorized by the deep learning diary er published by Lei Feng network, for reprint please contact the original author, and indicate the author and the source can not be deleted.