How to correctly enter data into the database?

Question

PostgreSQL

There is a need to pull updates from a third-party service via REST-api and bring them into the local database, in other words, just update the local database in real time.
Unfortunately, the third-party service does not have the ability to "subscribe to updates", but you can receive data for a certain period of time or the last n records.
The questions are:
1. What pattern should be followed?
2. This is a very idiotic idea: to implement a Watcher that will constantly send requests and check if there is a newly received data in the database?
3. What is the main idea of ​​solving such "Data Update" tasks? What should not be done, and what is the opposite?
Thanks a lot in advance for any advice.

Answer 1

Since local data is requested through its API, I would implement it in a similar way to how caching works. When data is requested from your API, a check is made to see how long ago the data was updated. If it is a long time ago (adjusting the time at your discretion), then a request is made to a third-party service and the data in the local storage is updated and the time of the last update is saved. The disadvantage of this approach is that the first request in a certain period of time will be processed longer due to all the operations.
If not critical, you can update the data in local storage after processing the request, giving the client "stale" data to avoid this pause. With the next request, the client will already receive up-to-date data, because By this time, they will most likely have time to update.
But there are more nuances to consider to solve your problem, this is just a general answer based on the information in the question.