What bundle will help with BIG DATA?

Question

MySQL

Colleagues, I need a brainstorm, since I'm not exactly an IT, I have general ideas.
There is an orgomny array of data in the form of a SQL database (there can be 100+ such databases). The structure of the database is identical, only the content differs. Task:
Given
Each record has its own ID
ID in the databases can and will be intersected (not always)
Comparison of records by ID is required
Idea
There is a certain resource with the necessary data. This resource serves for market analytics. Each page of the site (id) contains archival metrics. Pulling out data is not a problem, you can collect metrics. The task is to make a harvester for tracking key metrics for each ID. For example, I collected data a dozen times, sifted through "something" (this is what I'm actually looking for) and got a list of changes.
Simple example
Purchase and sale platform. Each page (ID) contains a label "Bought n-times", "Bookmarked n-times", there are millions of such pages (id). I have a cast on my hands, after some time I start re-collecting information. As a result, there are two casts, a comparison of values ​​\u200b\u200bfor ID is required.
Ideally, this is how:
I opened an application (on the same angular) of a table type with filters where there is an ID comparison function
| date | Metric 1| Metric 2 | DB Name
01 | 01.03 | 56 | 89 | name_bd_1
ID | date | Metric 1| Metric 2 | DB Name
01 | 01.04 | 101 | 133 | name_bd_2
------------------------------------------------ ---
ID | date | Metric 1 | Metric 2 | D.B. Name
01 | 01.04 | + 45 | + 44 | name_bd_1 + name_db_2
+ all possible filters, for example, by the Date column, combination \ comparison of more than two bases
Tell me how it is called correctly, since it doesn’t even look like a technical specification. How much would it cost, and how long would it take?

Answer 1

in general, this is called hadoop, there are frameworks like Flume for processing, this is if you play map reduce slowly,
if you hurry, you can process other options, up to the newfangled Clickhouse