Various approaches have been suggested for the regularization of neural networks, including the well-known Dropout and Dropconnect, which are simple and efficient to implement and therefore have been widely used. However, there is a risk of loss of well-trained weights when dropping nodes or weights randomly. In this paper, we propose a regularization method that preserves well-trained weights and removes poorly trained weights. This was motivated by the observation that the trained weights become further trained. We define these as eager weights whereas the opposite as lazy weights. On every weight update, the distribution of the changes in weight values is examined, and the lazy weights are removed layer-wise. The results demonstrate that the proposed method has a faster convergence rate, avoids overfitting, and outperforms competing methods on the classification of benchmark datasets.
ACKNOWLEDGMENT The authors would like to gratefully acknowledge supported from the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1A2C2003474), BK21 FOUR program of the National Research Foundation of Korea funded by the Ministry of Education (NRF5199991014091) and the Ajou University research fund.
