Lazy Node-Dropping Autoencoder - Lee, Jaewon; Kim, Myung Jun; Shin, Hyunjung (Proceedings - 2023 IEEE International Conference on Big Data and Smart Computing, BigComp 2023, 2023)
Autoencoders are widely used for dimensionality reduction nonlinearly. However, determining the number of nodes in the autoencoder embedding space is still a challenging task. The number of nodes in the bottleneck layer, which is an encoded representation, is estimated and determined by users. Therefore, to maintain embedding performance and reduce the complexity of the model, an indicator that automatically selects the number of bottleneck nodes is needed. This study proposes a method for automatically estimating the adequate number of nodes in the bottleneck layer while training the model. The basic idea of the proposed method is to eliminate lazy nodes which rarely affect the model performance based on the weight distribution of the bottleneck layer. Since the proposed method takes place in the learning process of the autoencoder, it has the advantage of accelerating the training speed. The proposed method showed better or similar performances in classification accuracy.
Keywords: Auto encoders; Dimension estimation; Dimensionality reduction; Embeddings; Learning process; Modeling performance; Neural-networks; Performance; Performance based; Weight distributions; Autoencoder; Dimension Estimation; Dimensionality Reduction; Neural Networks