Continual learning aims to adapt model parameters for new tasks while preserving knowledge from previous tasks. Recently, prompt-based learning has leveraged pre-trained models to facilitate learning new tasks through prompts, avoiding the need for a rehearsal buffer. While this approach has shown exceptional results, existing methods rely on a prior task-selection process to choose suitable prompts. However, inaccuracies in task selection can adversely affect performance, especially when dealing with a large number of tasks or imbalanced task distributions. To overcome this challenge, we present I-Prompt, a task-agnostic method that focuses on the visual semantic information of image tokens, thus eliminating the need for task prediction. Our approach features semantic prompt matching, which selects prompts based on the similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens at intermediate layers. As a result, our method delivers competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Furthermore, we demonstrate the effectiveness of our method in various scenarios through extensive experiments.
