Interpretability issue?

Question

Neural networks

Hello, I am writing a scientific work on the topic of human and artificial consciousness with an emphasis on the biological section of science. Many sources describe the so-called "problem of interpretability", I quote: "After training a neural network, we have practically no way to determine on the basis of what the neural network makes a particular decision (and even more so to influence this decision). It is often said that a neural network is it's a black box. Is it true? Is it true that there is no way to determine how any neuron, weight or bias affects the final result? Maybe there is some algorithm at the concept stage or some article that solves this problem?
Thanks in advance for your time!

Answer 1

Is it true that there is no way to determine how any neuron, weight or bias affects the final result?

This is not a problem - all weights are known. The problem is precisely to understand why the weights are the way they are.

Answer 2

Yes it is. A neural network is difficult to interpret because it is complex. Even in simple cases, a neuron is several layers of tens or hundreds of neurons, with each next layer depending on the previous one, and in addition, activation functions and dropouts are applied between the layers. One of the latest approaches to interpreting neurons is LRP (layer-wise relevance propagation). So google something like "neural network interpretation with lrp method". It is expected that all serious sources will be in English and filled with matan.