Activation-based cause analysis method for neural networks - Kim, Yong Gyun; Koo, Hyung Il (IEEE Access, 2021)
As neural networks are ubiquitous in the real-world environment, we encounter a variety of situations in which neural network-based systems fail. To address these failures, we need to identify their root causes. However, due to the black-box nature of neural networks, it is considered a difficult task to explain/understand internal functions, and cause analysis is usually conducted based on personal experience. To alleviate these problems, we propose a method to compute an element-wise contribution of inputs on current decisions. To this end, we focus on the physical meaning of neuron activations, and formulate an optimization problem that finds partial activations that support current decisions. Then, by accumulating these partial activations in the backward direction, we evaluate the contributions of each element in an input vector to the current decision. Experimental results have shown that the proposed method outperforms conventional methods in terms of cause localization for a range of failure scenarios.
Keywords: Conventional methods; Failure scenarios; Network based systems; Optimization problems; Partial activation; Personal experience; Physical meanings; Real world environments; explaining classification; interpretable machine learning; Multilayer perceptron