Channel Propagation Networks for Refreshable Vision Transformer

Abstract

In this paper, we introduce the Channel Propagation method, which aims to increase the channels of the Vision Transformer systematically. Skip connections are commonly acknowledged as a propagation approach that improves the stability of the performance in Vision Transformers. Nevertheless, it is important to note that these skip connections may give rise to the problem of over-smoothing, wherein similar features are represented in multiple layers. To tackle this matter, our proposed approach for Channel Propagation in Vision Transformers retains the present signal information while concurrently propagating location-specific signals in a newly introduced channel dimension. On the other hand, the proposed Channel Propagation method effectively maintains the integrity of identity representation while simultaneously including patch-wise location-specific supervision by introducing a new channel dimension. The inclusion of this approach in Vision Transformers mitigates the issue of over-smoothing while also improving the performance of visual recognition tasks. In our experiments, we confirm that the proposed method is effective for various visual recognition tasks. Specifically, our method demonstrates enhanced performance when implemented on Vision Transformer models; the classification accuracy is increased considerably for plain and hierarchical architectures on the ImageNet dataset.