How to correctly organize communication between services of a mixed type and notify the user about the completion of the process?

Question

Python

I am involved in the creation of a system consisting of several clients and services. Plugins for desktop applications and WebUI act as clients through which the user interacts with the system . I will not indicate the entire list of services, I will list only those that are important to indicate the situation:

• Kong API Gateway behind which everything is hidden;
  
• The database service ( MongoDB ), written in FastAPI , is a regular REST API ;
  
• Information processing service, now it is also FastAPI , generating Celery tasks in which I request files from the database service and process them;
  
• Celery service that accepts tasks and executes them (using Redis as a broker );
  
• Service written in NestJS serving WebUI .
  

In the current version, the standard workflow is as follows:

1. The user using the plugin sends an HTTP request with data to the database service;
   
2. The database service validates the information, if everything is fine, it writes to the database;
   
3. After creating a new entity in MongoDB , a request is sent from the database service to the information processing service (to be converted to the default format). No response expected;
   
4. The information processing service receives a request and if no one has already performed the same operation before, it creates a Celery task;
   
5. Celery service performs the conversion, puts the result into the repository, and that's it;
   
6. The user opens the WebUI and sees the entity stored in the database ( the NestJS service makes a request to the database service), the view window displays the model obtained during the conversion process (the NestJS service makes a request to the information processing service);
   
7. If the user wants to convert an entity not to the default format, but to some specific one, then he makes a request to the endpoint of the information processing service;
   
8. The information processing service creates a Celery task and returns its task_id to the user in response ;
   
9. The NestJS service periodically asks the processing service for information about the state of the Celery task;
   
10. When the conversion ends and the status of the task changes to SUCCEEDED , the user requests the result from the data processing service.
    

It was decided to move from a similar system that uses HTTP requests for communication between services to communication through a message queue. NATS Streaming was chosen as a tool for implementation . Now, at the stage of thinking through the changes, questions arise that, due to my inexperience, I cannot objectively answer myself.

For example: If, after creating an entity in the database, I place an event message in the NATS common bus , then I can easily perform the necessary actions in the new service and also, at the end of them, place a new message notifying about the end of the processing process, everything is fine, BUT ... What if someone wants to convert an entity to another format? What then? Let's say I leave RESTdata processing service to initiate this procedure, then in it I should put the event message “conversion to such and such a format was requested” in NATS instead of creating a Celery task ? The service “listening” to such “events” will start converting and… And… so what? The user made a request via HTTP , the endpoint created an “event” and replied that they say, wait, but why wait? How to notify the client that the conversion is over or that an error has occurred? In the current version with Celery , there is a task_id by which you can check the current state of affairs, but what about the new system? There is an idea that to combine REST services with services communicating through MQnot a very good idea. But then what to do? Can I create a separate service that keeps a constant connection with the client via a WebSocket connection, like NestJS for WebUI only for other clients? The user initiated the conversion, the service listens to such events and, when responding, notifies the user via WS ? What then to take for the implementation of such things? Maybe even let everyone through the same NestJS service that serves WebUI , but why, even with plugins, keeps the connection ?! But then the question of authorization arises, as in NATSfilter only the events needed by the current user? If N users connected, each started converting their entity from the database to the desired format, N events were created in NATS , of the form entity.converted , then all services listening to the event bus for the completion of the conversion will accept ALL events and you need to manually serialize the body of each message and see if my essence has been converted?

As a result, the main question is, what can I read so that everything falls into place in my head?

Answer 1

REST is synchronous, message queue is asynchronous. If you switch to async, for HTTP you will have to implement WebSocket, or long-poll, or crutch poll once a second, for example. (Fu!)
Tasks to identify, so that several who request one, get the only one.
Check out the Pub-Sub pattern. For example, the one who sent the task "I want data X in Y format" subscribed to the messages of the "Gotovchenko" channel and waited for his "XY" in it.
It is convenient for me to schedule messages and events in an asynchronous system in the online editor of UML diagrams. Participants of the processes are at the top. Down goes "time". And interactions, who sent what to whom.

Answer 2

Only 3 components:
1. Event generators - from different systems.
2. Event listeners - those who are interested in the status of systems.
3. The event queue manager for a specific user (one of the event listeners is the same).
Delivery from the manager to the browser is another matter altogether!
There are a lot of options: manually, js-timeout (or other client event), long-polling and websocket.
About UML - Sergei Sokolov has already said everything correctly.