How to write information from the database to files as quickly as possible?

Question

MySQL

Hello!
Task: there are more than 7 million entries in the database table. records. It is necessary to write samples from this table into files of different formats (doc, txt, exсel). Samples can range from hundreds of lines to several million. If an entry in the database is changed, you need to update all the files in which information about this entry was found (the maximum update delay time is a couple of minutes).
I see the solution to this problem in the queues. Those. a task is created in the queue for the corresponding selection. Further, in the task, the selection itself is made in portions, 1000 records from the database and written to several files.
How viable is this approach? Tell me, please, maybe there are patterns for such situations when you need to extract information from the database as quickly as possible and scatter it among the files.

Answer 1

You will have to keep a lot of files somewhere, depending on each particular line. When it is updated, update the corresponding set of files.
If several lines are updated at once, then take the intersection.
File update tasks can indeed be done through a queue with a fixed bandwidth and possibly a stable timer.

Answer 2

xml file can be given by a script generating it dynamically.
for example, the site.com/file.xml url
may not be present when requesting this file url, and the web server will run, for example, a php script that will generate and return it as xml. also this script can cache the file to give faster.
a very common tactic for caching image thumbnails, for example.
for example, there is a path
site.com/media/thumbnail/150x200/4564564.jpg
when you request it, if there is no file, the script will run, make a thumbnail with the requested permission and put it in the folder at the same path /media/thumbnail/150x200/4564564. jpg
on the next request, the file will exist and the PHP script will no longer run.
when the picture becomes irrelevant, for example, it changes, the thumbnail will be deleted and the next request will generate an up-to-date one again.
I think it will be the best solution for you too.