Learning Neural Networks without Lazy Weights - Lee, Dong Gi; Cho, Junhee; Kim, Myungjun; Park, Sunghong; Shin, Hyunjung (Proceedings - 2022 IEEE International Conference on Big Data and Smart Computing, BigComp 2022, 2022)
Various approaches have been suggested for the regularization of neural networks, including the well-known Dropout and Dropconnect, which are simple and efficient to implement and therefore have been widely used. However, there is a risk of loss of well-trained weights when dropping nodes or weights randomly. In this paper, we propose a regularization method that preserves well-trained weights and removes poorly trained weights. This was motivated by the observation that the trained weights become further trained. We define these as eager weights whereas the opposite as lazy weights. On every weight update, the distribution of the changes in weight values is examined, and the lazy weights are removed layer-wise. The results demonstrate that the proposed method has a faster convergence rate, avoids overfitting, and outperforms competing methods on the classification of benchmark datasets.
Keywords: Deep learning; Layer-wise; Learning neural networks; Neural-networks; Overfitting; Regularisation; Regularization methods; Simple++; Weight update; Weight values; Deep learning; Neural networks; Overfitting; Regularization