What are the best tools to use to create a comparison service for large amounts of data?

Question

Database

Good day dear developers. Once again, there is a great task of creating a smart and working IT service. Let's go straight to the problem that has arisen:
There are two data stores: a certain database of the Billing System (system X) and a database of the system for providing licensed access to paid content such as Verimatrix (system Y).
When a customer makes a purchase of a paid service (for example, purchasing content from a mobile application, interactive set-top box or website), information about the purchase is registered in system X and a command is sent from system X to system Y to open access to the corresponding piece of content. There are intermediate links between systems X and Y (command converters, queues, etc.) that can lead to a violation of data consistency in the systems under consideration. The task is:once a day, upload files of the same structure (client ID, list of available services) from systems X and Y and compare these files . In essence, the task is to create a service Z that has the following characteristics:
1. service Z must be taken out of systems X and Y ;
2. service Z must accept large files as input from each of the systems X and Y, compare files and display the discrepancy result ;
3. service Z must be able to send commands to the command queue ;
4. service Z should signal other services if the discrepancy indicators are out of range of acceptable values .
Maybe someone faced similar tasks and will share experience and information on the tools used? I will be glad to receive advice and speculate a little or get links to ready-made tools so as not to reinvent the wheel.

Answer 1

The file is maximum 12 GB, this is common with all city databases, you can organize that each database sends files separately (36 databases in total), then the files will be approximately 350 MB in size.