What is the reason for the drop in application performance?

Question

Video

There is an application, the essence of which is the processing of a video signal from several IP cameras. The application receives frames in the form of a bitmap, some manipulations are performed with them, then the frame is drawn on the screen and sent over the network. The process is constant in the form of several independent logical threads.
So here's the problem. When working with a single video stream, the performance of the algorithm allows processing an hd 60fps video stream, and when testing individual parts, it accelerates to >100 fps. But, when you start processing several video streams simultaneously, there is a drop in performance, fps drops on each logical thread - the decrease is uniform, i.e. video processing 25fps, one stream is processed normally, 2 - fps 22-23, 3 - 17-19, etc. When measuring the speed of execution of critical parts of the code - it is clear that everywhere there is a slowdown - the main processing functions are copying memory between bitmaps and compressing them into jpeg.
There is a suspicion of a garbage collector - because. A lot of memory is allocated, but I'm not sure exactly. the debugger shows that % time in gc does not exceed 10% .
Help find the cause. thank.
ps1. with all this, the hardware is not loaded at 100%
ps2. when starting video stream processing in different physical processes, i.e. several instances of the program - no problem is observed, i.e. there is no drawdown.
ps3. after some manipulations and optimizations, I came to the conclusion that the performance drop occurs only on the virtual server, both with 16 and 48 cores. - i.e. The CPU is not loaded and the process is running at low fps. also works bare ffmeg when run from console with equivalent setup.

Answer 1

How do you process images, display them, and what functions do you use? The fact is that C# is not very well suited for such things, and the standard mechanisms for working with individual pixels are very slow there. To work with images, you should use at least unmanaged code, and in your case it is even better to put all the functions of processing and displaying the image in a separate module written in C / C ++ or something else like that.

Answer 2

You need to explore how threads work using for example the Concurrency Visualizer . It is possible that locks occur, due to which your cpu utilization drops.

Answer 3

the problem was solved,
the main problem was in setting up the network - the bandwidth did not allow receiving the signal in the required volumes.
secondary - garbage collection, decided by reusing buffers.