Understanding and explaining convolutional neural networks based on inverse approach

Abstract

Interpretability and explainability of machine learning systems have received ever-increasing attention, especially for deep neural networks. In the case of convolutional neural networks (CNNs), their properties are usually explained by generating local explanation maps (e.g., visualizing the contribution of individual pixels to a given prediction). In this paper, we propose a new framework that analyzes the inner workings of CNNs in terms of neural activations. To be precise, we consider a forward-pass as sequential activations of neurons and develop its inverse process, so that the inverse preserves the physical meaning of neuron activations. Our inverse process is formulated as a constrained optimization problem, and we solve the problem with the gradient projection algorithm. The proposed approach can provide equivalent visualization results to several conventional methods, and thus can be a reference tool for CNN visualization. Also, the attributions generated by our inverse method yield the state-of-the-art deletion scores and visualize the contribution of colors as well as shape features.