Activation-based cause analysis method for neural networks

Abstract

As neural networks are ubiquitous in the real-world environment, we encounter a variety of situations in which neural network-based systems fail. To address these failures, we need to identify their root causes. However, due to the black-box nature of neural networks, it is considered a difficult task to explain/understand internal functions, and cause analysis is usually conducted based on personal experience. To alleviate these problems, we propose a method to compute an element-wise contribution of inputs on current decisions. To this end, we focus on the physical meaning of neuron activations, and formulate an optimization problem that finds partial activations that support current decisions. Then, by accumulating these partial activations in the backward direction, we evaluate the contributions of each element in an input vector to the current decision. Experimental results have shown that the proposed method outperforms conventional methods in terms of cause localization for a range of failure scenarios.