Why does the stream start to slow down?

Question

linux

Greetings, the task was to develop a server for online broadcasting (stream), googling found the solution nginx-rtmp-module , it works stably, there are no problems with it, the problem is different.
And so there is an HLS stream via OBS and a server where it all works and is distributed, here is the nginx config .
There is a problem with a large number of users connected to the stream, if in a situation where a couple of dozen users are watching the stream, .ts files are given in about 200-300ms, and if 500-1000 users connect to the server, then the return is stretched from 5 seconds to 20 seconds
I thought that the problem was the lack of bandwidth, each .ts file weighs about 2mb, in general, I created a bunch of servers with this nginx config:
nginx.conf

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=pagecache:50m inactive=50m max_size=1G;

default:

proxy_cache_valid 200 301 302 304 5m;
  proxy_cache_key "$request_uri";
  proxy_hide_header "Set-Cookie";
  proxy_ignore_headers "Cache-Control" "Expires";
  proxy_cache_use_stale error timeout invalid_header http_500 http_502 http_503 http_504;
  proxy_cache	pagecache;

location ~ \.ts$ {
    proxy_pass              http://сервер-стрима;
    proxy_set_header        Host             $host;
    proxy_set_header        X-Real-IP        $remote_addr;
    proxy_buffering on;

    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Cache-Key $request_uri;
  }

And on the client, we already randomly access one of the cache servers
in this situation, the ability of the stream to work with a large number of users increases, but this also works up to a certain number of users, for example 5 thousand users
. New .ts files are added to it, it must be served directly from the stream server, then it cannot be cached, and with a large number of users, its return from 40ms increases to several seconds.
Please help with creating the correct server structure and caching, I raise the servers in vscale.io
I want to note that it works perfectly if you use cloudflare and configure page rules to fully cache .ts files (cache-level: cache everything), but after a while cloudflare disconnects from the zone, since apparently it does not allow caching .ts on a free plan or at all files, if this is solved using cloudflare, it would be more convenient, another company raises a similar stream and they use cloudflare, but for some reason it doesn’t work for us.
Before that, I used aws cloudfront, it is not stable and is very expensive, the stream can take 2 hours and it is more profitable to just temporarily raise a bunch of servers from a backup on some cloud service and pay very little money.
In general, please tell me how to solve the problem with the stream freezing, with using nginx or cloudflare

Answer 1

Then there is a problem with returning the playlist of the .m3u8 file, since new .ts files are added to it, it must be given directly from the stream server, then it cannot be cached

m3u8 can be cached.
The master playlist can be cached, for example, for a minute - the number of options (qualities) does not change often.
A playlist with ts(chunklist) can be cached for the length of a chunk - if the chunks are 10 seconds, then calmly cache them for 10 seconds.
You can put a caching front (with cache_lock enabled) in front of nginx with an rtmp module, cache on a ramdisk, cache chunks, caching time depends on the size of the chunklist.
Why is hls_fragment set to 2 and 3 seconds?
I advise you to do 5.
How is this implemented?
In general, a completely working scheme for http streaming is: a
packetizer (in your case, nginx-rtmp)
2 origin-shield servers - for fault tolerance and to protect the packetizer from a large number of requests
N edge servers
We distribute 10 gigabit from one ezhda (there is not enough channel ) ...
The maximum was what to squeeze out of them - 40 gigabits, but this is only inside their network, there are no such channels outside.