How to wait until the data is written to ETL before giving back to the client?

Question

Database

Hello.
The question that has been tormenting me lately is how to correctly expect data on the websocket side.
Notifications about new events come at the time of adding data, via ZMQ I send a string with the name of the changed field to WS.
The ETL system is quite large (crutch) and it may take some time before adding to the database. Sometimes large indicators are recalculated and you have to wait tens of minutes. Sometimes just little data comes in and until a sufficiently large package is collected, nothing simply gets into the clickhouse.
Now I set an artificial expectation - I write down how long I waited for this field and take the average value.
+ I contain a hash table for subscriptions of ws clients with hash sums of past results.
But it all looks rather hacky and does not work very well, there is often a data failure for some reason.
How did you implement this in your projects?

Answer 1

In fact, it is not very clear what is happening with you, but I want to tell you a few facts:
- the event system assumes that a synchronous request gives the user only the task ID and a message that it is being processed
- when processing is completed, either back through the web socket, or through short-pooling/signal-r you get a change in the task status
- as for clickhouse, they have eventual consistency declared, so it makes sense to send batches only if you have cost optimization in the cloud