Finding new or changed records in a large dataset

Question

PHP

There are a number of platforms, the data from which is transferred to the final platform.
A large number of platforms are databases, some of the platforms provide access via API.
The total number of records is about several million.
Also, the data on the source platforms may change - you need to track changes and update the data on the target platform.

Suggest how best to implement the mechanism for uploading new data, tracking changes and updating?

Answer 1

The conditions are too vague, much will depend on the specific API. Here is one of the options for solving a similar problem for mysql tables www.percona.com/doc/percona-toolkit/2.1/pt-table-checksum.html
maybe some ideas will be useful to you.

Answer 2

Native means of database replication - IMHO more efficient.
If the database can send signals to the application (I met it in Interbase), generally a buzz.
If the databases are heterogeneous, then you will have to write the entire replication kitchen from scratch on your knee. With their bikes and freshly opened old rakes.