Why do streaming programs start to slow down after disconnecting from the server?

Question

fmpeg

There is a virtual Windows Server 2012 R2 running in VPC Selectel.
I run ffmpeg on it to broadcast RTSP from surveillance cameras to YouTube.
A problem constantly arises - when broadcasting, after several hours, the stream starts to lag behind real time (after three hours, the lag turns out to be 20 minutes). However, as soon as I connect to the server via RDP, the video instantly jumps to the current moment. As a result, time gaps appear on the recording after the stream.
To avoid this, you have to connect to the server every half an hour.
Previously used OBS Studio, the problem was then.
What could be the reason and how to fix it?