How to change interdependent objects at the same time?

Question

Java

I'm trying to create something similar to a neural network, and there was a problem. There are ten neurons, information from the output of each of them goes to the inputs of all the others. And if they are counted sequentially, as usual, a problem arises. Recalculating the value at the output of the first neuron and starting to recalculate the value of the second neuron, we get that the value for the second neuron is calculated incorrectly, since it takes the new and not the old value of the first. This process continues like a snowball. How to calculate the values ​​of neurons for her particular state?
The only option that I see is to remember the old value of the neuron, recalculate the new one, remember the new one, force the old value into the neuron. So, in turn, count all the neurons, and then assign them the memorized new values. But I don't like this approach. Can it be done in some other way? Stubbornly it seems that this should be possible if executed in many threads, but is absolutely not familiar with multithreading.