What is the correct way to load a large amount of data into the database in laravel?

Question

Laravel

Good afternoon, I am creating an online store on laravel, the goods to which will have to be imported from VK via the Market API. The administrator in the control panel presses the "Import" button and all goods from the VK group must be recorded in the database. There is a lot of data, therefore, the server will not be able to "swallow" everything in one sitting. In the Shopify online store builder, product data from a file is imported within 15 minutes, that is, background loading is organized. How can this be done in laravel? I read about queues in the docks, but I didn’t quite understand how to use them correctly in this task. Who knows, please help.

Answer 1

You click on import, send a regular HTTP post request (docs), create a job in the handler and throw it into the queue (docs), write the import logic in the job (maybe at least an hour, at least two, at least ten years it will be imported - the main thing is that the server does not fall, and don't forget to write a queue name other than default).
Actually everything. Then you either raise the queues with the supervisor (docs), or fasten the horizon (docs) and, of course, some adequate driver for these queues, such as rabbitmq or redis (and turn on the corresponding driver (docs)).

Answer 2

How to work with jobs, wrote above. If you want to optimize the very insertion of data into the database, then you can write data to a file in Job, and then load it into the database via load data infile. It will be significantly faster.