P
P
Philip Bondarev2020-12-10 21:02:48
Message Queues
Philip Bondarev, 2020-12-10 21:02:48

How to properly start a Python (micro)service to work with NATS?

What I have now:

  • Database entry service (validates incoming data, looks for duplicates) [ FastAPI ]
  • Data conversion and transformation service (takes data from the database service and creates Celery tasks ) [ FastAPI ]
  • Service with Celery on board + Redis as a broker to it
  • Front service (not me sawing, xs what's there), which interacts with the first two apish [ NestJS ]
  • The front itself [ Angular ]

Accordingly, as an example:
After writing the data to the database, I pull the endpoints of the conversion service from the database service to initiate the conversion to the default formats. If another format is needed, then the user presses the necessary buttons from the interface, and the front service is already pulling the handles of the conversion service. All this needs to be logged, end-to-end tracing will be needed in the future, etc. With the growth of functionality (and, consequently, an increase in the number of services), interaction will increase, connectivity will increase. You will have to handle cases of falling off of part of the services in the code, etc. As I understand it, if we switch from manually pulling endpoints to an event-based model of service interaction, it will be possible to bring the system to the following form:
  • Database write service
  • The service for manual conversion initiation, conversion - an apish with no logic for creating tasks, just endpoints that create events by type - a conversion to such and such a format was requested.
  • Conversion service
  • Data transformation service
  • front service
  • Front

Then, having a Pub/Sub bus in the form of NATS , I can do everything as follows. The user sends data to the database service, after validation the data is written to the database and a message (event) with information about the new data is pushed to the NATS subject . For this event, we have subscribed conversion services and the front service. Accordingly, the conversion should start in the conversion service, and the user's light will blink on the page - the data is recorded, everything is ok.
The conversion service, at the beginning of the conversion, creates an event - started. A front is subscribed to this event - the corresponding light flashes to the user and further in the same vein. At first glance, everything is simple, but upon further reflection, I have questions.
When creating RESTI use FastAPI for services, there is a customized Docker image for it, in which it is enough to put the application in a certain place and wrap everything up ... Packed in such images, the apishki are able to serve a huge number of users. However, from what I described above, it is clear that not all services are REST APIs . Data conversion and transformation services are, in fact, just scripts placed in separate containers, and I have no idea how to properly organize it all. If I write a script that just waits in an infinite loop for messages (events) in the subject 'e NATS, then how to run it all in a container later, so that the script can serve the n-th number of events at once. Is it necessary at all? Are there examples, articles, videos on the topic? All I find is the next "How to run Flask in docker!"

As one of the assumptions:
In the conversion and conversion services, create all the same Celery tasks and keep the service with its workers in parallel, in which you can also push events with information about the progress of the process to NATS .

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question