Do you need layer synchronization inside a neural network?

Question

Neural networks

Those. stop data processing on the next layer until all the neurons of the previous one have worked out. After all, the input data does not go to all inputs at once, but is entered sequentially in a cycle.

Answer 1

If you multiply by a column and consistently calculate layer by layer, there is no such problem.
If you calculate in parallel, for example on the GPU, then yes, synchronization is performed by hardware, by means of the GPU.