Being an informal communication source, social media text is susceptible to several intentional and unintentional textual variations. These variations lead to various out-of-vocabulary (OOV) words, making social media text processing more challenging. This work analyses and discusses such challenges by providing a detailed overview of different sources of intentional and unintentional OOV words and associated challenges. We provide a detailed survey of pre-processing techniques, including traditional and application-specific methods proposed in the literature to handle intentional and unintentional textual variations, while highlighting their pros and cons. The paper analyses the implications of text normalization (standardization) in different social media text-processing applications. Moreover, the paper provides an overview of the recent challenges and trends in handling social media textual variations, and it is expected to provide a baseline for future research.
This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the Artificial Intelligence Convergence Innovation Human Resources Development (IITP-2024-RS2023-00255968) grant and Grant RS-2021-II212068 (Artificial Intelligence Innovation Hub), supervised by the Institute for Information & Communications Technology Planning & Evaluation (IITP), and also by the National Research Foundation of Korea(NRF) grant (No. NRF2022R1A2C1007434). The publication fee has been paid by the BK21 FOUR program of the NRF of Korea, funded by the Ministry of Education (NRF5199991014091).
