Answer the question
In order to leave comments, you need to log in
How to store temporary entities?
Hello.
There is a web application with the following functionality:
The user can create an entity in the database (PostgreSQL), let's call this entity: "module". The module can then be accessed via the API. While there has not been a single request to the module, it is considered temporary and will be deleted in a day. As soon as the module has had at least one call, it is "fixed" and remains active for life (until the user deletes it).
There are 2 different types of module requests and many large partners send 2 requests at once. By "time" I mean a time difference of a few milliseconds. The average query execution time is 100ms, so these queries are executed in parallel.
The essence of the problem.
The number of real modules (which were accessed at least once) is approximately 1% of all created ones. In order not to litter the database, I tried to create a temporary module in redis and then, when accessing it, transfer it to the database. This idea failed, because in the case of two requests, while one request pulled out the module structure from redis and created an entity in the database, the second request did not find an entry either in redis (since the first request had already deleted it from there) or in the real database ( since the first request has not yet committed the transaction).
At the moment, a temporary module is created immediately in the main database, but I don’t like it, since it is currently being created at about 20 thousand modules per day, and in the future it will be 400-500 thousand and more.
Questions.
What is the risk of daily adding and deleting a large number of rows to a table, and is it possible to not worry about it on such relatively small volumes and start thinking about a solution when the number of daily records approaches a couple of million? It should be noted that there are still constant selections from this table, so a good access speed is needed.
How can you competently solve the problem if the goal is to write only "live" modules to the main database? I don’t want to localize the entire table for a while while one query takes a record from the radish and transfers it to the database, because it will be a lock of the entire table, not just one record ..
PS Changing the logic and making 2 types of access through one query is not an option from the word " at all". This logic is not set by us and no one will change it for us :)
Thank you.
Answer the question
In order to leave comments, you need to log in
I would do a cron to clear the table of entities that haven't been accessed for a set amount of time. Naturally, it is necessary to fix the fact of the appeal somewhere. Entities to store in one base.
Or a queue - when creating an entity, put a task in a pending queue, which in a day gets into the consumer and is processed.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question