Visualization algorithm based on FDR control testing for dimension reduction of textual data

Abstract

Purpose: Visualizing relations of textual data requires dimension reduction to increase the interpretability of output. However, traditional dimension reduction methods have some limitations, such as the loss of feature information during extraction or projection in dimension reduction and uncertain results due to the mixture of word labels. In this study, we develop the textual data visualization algorithm using statistical methods to present statistical inferences on the data. We also construct the algorithm in a way that the user can analyze textual data easily. Design/methodology/approach: Unstructured data, such as textual data, is sensitive to choosing analysis methods. In addition, textual data is generally large-sized and sparse. Considering such characteristics, we applied latent Dirichlet allocation to separate data to minimize the loss of information, and false discover rate (FDR) control to reduce dimension in a statistical way. Findings: The relation of textual data can be derived in a one-click way, and the output can be interpreted without background information, with separated topics. Originality/value: The algorithm is constructed based on the Korean language. However, any language can be used without linguistic information. This study can be an example of usage and flow, which using not well-known dimension reduction methods can replace traditional methods.