Lazy Node-Dropping Autoencoder

Abstract

Autoencoders are widely used for dimensionality reduction nonlinearly. However, determining the number of nodes in the autoencoder embedding space is still a challenging task. The number of nodes in the bottleneck layer, which is an encoded representation, is estimated and determined by users. Therefore, to maintain embedding performance and reduce the complexity of the model, an indicator that automatically selects the number of bottleneck nodes is needed. This study proposes a method for automatically estimating the adequate number of nodes in the bottleneck layer while training the model. The basic idea of the proposed method is to eliminate lazy nodes which rarely affect the model performance based on the weight distribution of the bottleneck layer. Since the proposed method takes place in the learning process of the autoencoder, it has the advantage of accelerating the training speed. The proposed method showed better or similar performances in classification accuracy.