Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.
Funding: This research was funded by the European Regional Development Fund (Grant project number 20201434 and NYPS 20204318), and this research also supported by the new faculty research fund of Ajou University (S-2020-G0001-00478).This research was funded by the European Regional Development Fund (Grant project number 20201434 and NYPS 20204318), and this research also supported by the new faculty research fund of Ajou University (S-2020-G0001-00478). The authors would like to thank Karan Mitra and Saguna saguna from Luleå University of Technology for helping us improve this article with their comments.
