What algorithm for unloading data from the server is better?

Question

Software design

The server contains a database with clients and information about them.
The manager of the company has software that connects to the database and downloads everything to itself via sockets.

When connecting to the server, a list is created for sending, which is divided into parts (50 clients per message) and sent to the manager with a pause of 1 second. Small data packets are sent faster than one big one.

How can such a problem be solved in large projects? What is the best way to download typical data from the server so that all records reach?

Answer 1

In large projects, there will most likely not be such a task directly. Even sometimes, inverse problems arise - "how not to let the manager steal the entire client base")
And the rest - in general, "all at once" is somewhat unnatural. One option is reports. That is, the formation of a report according to the desired template and criteria. Moreover, the criteria can be different - either "active this month", or "not buying for half a year", "accumulated amounts and average monthly volumes".
Well, "large" projects are also reliable communication channels - that is, even if a dozen gigs stretch - a failure is an emergency

Answer 2

Looking at what form they should be.
JSON -> 7z -> upload via HTTPS to multiple streams with resume as an option.
As a rule, a DBMS is not exposed directly to the Internet with a bare socket, an API (with HTTPS, Cert-pinning and key authorization) sticks out, which gives the necessary junk, compressing it into gz on the fly.

Answer 3

Replication and date kiosks, in normal databases it is right out of the box. And you can fine-tune what to send. And this is all at the database level without crutches and always works