How to effectively calculate new / left users of the Vkontakte group?

Question

ruby

I need to have up-to-date data on new and departed users from the VK group. I will be working with a lot of groups.
Initially, I get the id of all members of the VK group and save it in their database as separate records in a table group_memberswith fields group_id, user_id.
Periodically, I re-interrogate VK to update information about users. Checking all ids one by one is inefficient. I think that it is possible to compare the id array with what is in the database in order to first find those who are not in the database (new ones), and then those who are not in the VK response (gone). What is the best way to do this?
Because people constantly come/leave, then there will be large deviations in the arrays. Do they need to be stored somewhere if, for example, we are dealing with a group with 10 million members? Redis?
What is the most efficient way to do all this?
I use PostgreSQL, Ruby on Rails.
UPDATE
VK has a method executeso you can get 25000 members at a time. Callback API, unfortunately, connects only to my community, and I want to monitor competitors. I understand that it will be necessary to buy a proxy IP for speed.

Answer 1

I advise you to radically revise your user accounting system. Receive users at the time the group is connected in the same way as now. And keep records through the Callback API , where to subscribe to the subscription \ unsubscribe from the community. Then you will not need to periodically get users and compare array discrepancies. You will receive a specific user at a specific moment and find out whether he unsubscribed or not. This will simplify the logic, give the exact time of subscription/unsubscribe, but thereby increase the load on the server, since after each user action a request will be sent to the server.