How to synchronize a large amount of data?

Question

1C-Bitrix

Bitrix24, box. I am writing a module for collecting information from clients, which would then be sent for processing to a third-party service.
Problem:
Data volume, about 40 million records.
I'm trying to send this to a third party service as json. Through experiments, it became clear that it is easiest to send files in parts of 8 MB (a third-party service allows you to do this).
But I'm running into a script execution time limit.
While the idea came to mind:
Write a script, put it on the agent (Bitrix cron) and in this script: -
pick up some of the files
- write the last id from this piece of files to the table
- send
and so on in a circle until everything goes.

Question:
Does anyone have any ideas how to do this easier and faster?
Ps set_time_limit must not be touched. this code will be executed on the client + all scripts are run by Bitrix agents

Answer 1

There are a couple of options for how to do this:
- Stepper (the type of agents you wrote about)
- a cron file.
There is nothing wrong with performing a specialized task for which a cron file / cron script is suitable for performing in this way.
Moreover:
- In the box, the execution of agents on the cron can be configured, which solves part of your problem
- Bitrix in certain subsystems (for example, open lines) itself recommends additionally configuring cron jobs