How to do many checks?

Question

PHP

Sometimes there are tasks when you need some data from api, push it into the database, but at the same time check it first.
With small data, I read the entire table I need and work with it as with an array. And then I check.

But the problem appears when such a table grows and at the same time a lot of data comes in.
That is, reading the table is not a problem, but if there is not much memory on the server, then there is already a problem. You can't stuff everything into it.

Doing a check every time, directly in the database, it seems to me that it is also not very rational, because. a million data may come at a time and making a million read requests is not an option. Even if not a million, but several tens of thousands.

How do you solve problems like this?
It's just that you either need to make a lot of requests, or keep a lot of data in memory ...

Answer 1

to push in a DB, but at the same time at first to check.

What and with what?

With small data, I read the entire table I need and work with it as with an array. And then I check.

The worst possible option, of course, no matter what you compare there, is a complete akhtung.

But the problem appears when such a table grows and at the same time a lot of data comes in.

logical and expected

Doing a check every time, directly in the database, it seems to me that it is also not very rational,

It strongly depends on the check, as well as on the task as a whole, which you have not voiced, what you are checking there ...

How do you solve problems like this?

In different ways, a little specifics will greatly help you choose the right work algorithm.

Answer 2

Although right now I got the idea that it would be possible to store in the table, id with a unique value, and each time throw an exception if there is such an entry .... That is, mysql itself would give an error .....

INSERT ... ON DUPLICATGE KEY IGNORE

what if you do not need to check the uniqueness of a record in the table, but for example, what if, for example, the sum of click values ​​is greater or less than the sum of conversions?

But after all it needs to be done not during an insertion?

SELECT count(*) FROM `my_tbl` WHERE `date` BETWEEN(...) AND type='conversion'
SELECT count(*) FROM `my_tbl` WHERE `date` BETWEEN(...) AND type='click'

Well, if there is really a lot of such data, then you can look towards specialized solutions like clickhouse.