What is the best way to parse a database?

Question

Python

I make a request to the site database and get the following json response:

{
 "info": [
        {
            "i": "1234567", ### Уникальный идентификатор
        },
        {
            "i": "7654321", ### Уникальный идентификатор
        },
        {
            "i": "1957833", ### Уникальный идентификатор
        },
        ]
}

The bottom line is that there are about 25,000 such "Unique Identifiers", naturally each is one of a kind and is not repeated.

The database is updated very often, some identifiers disappear, some appear, the place of appearance is random (it can appear both at the beginning, and in the middle or at the end)

There is only one task: Find the best way to quickly find all the NEW values ​​added in the last time. (In other words, see what's new in the table and what's gone after a couple of minutes)

The task seems simple, but I've never worked with such a large amount of data before.
The thought immediately comes to mind just to add all these identifiers to your SQliteStudio database, and after ... just compare each identifier from the SITE BASE with my database? ... If there is no identifier in my database, add it to my database. If there is no identifier in the SITE BASE, but mine has it, delete it from my database.

But, comrades, am I shooting sparrows with a cannon?
Maybe there is a more sophisticated method or function than a simple search on 1 line and compare with my database?

Speed ​​is important, so I want to hear your opinions: do I have many solutions?

Answer 1

25000 is nothing. Load JSON, make a set of identifiers and make a difference with your set (store, in general, no matter where)