T
T
thethanosdaddy2021-05-03 11:14:37
Backend
thethanosdaddy, 2021-05-03 11:14:37

Service architecture for collecting and processing text data. We need healthy criticism. /?

Task:
Design the following service according to the description. There are N user workstations on which text data aggregators are installed. The data is sent to the servers and processed to extract the text for further full-text search. The service must scale.

I took the microservice architecture as a basis. In this case, clients only send data and do not expect any response. Similarly, intermediate services. Scaling in my scheme is assumed by increasing the performance of the most loaded microservices (with a change in their "weight" at the balancer), or by increasing their number. The web server at the end of the diagram is just an example of an end user. There could be anything.608fb06db46cc164879578.png

Answer the question

In order to leave comments, you need to log in

4 answer(s)
V
Vladimir Korotenko, 2021-05-03
@firedragon

Personally, I don't understand what clients are doing. Scrap the web and send data to the server? Or do they simply receive data, and the server aggregates and then gives it to other clients?

U
uvelichitel, 2021-05-03
@uvelichitel

database of full-text search services are they replicas? Are they completely synchronous, or is there a partitioning/sharding mechanism? This database can generally be considered a separate service (separate from full-text search services). The benefit of DB engines that have their own scaling mechanisms is enough in nature.

G
Griboks, 2021-05-03
@Griboks

I can see that you forgot the balancer in front of the web server.
Well, it turns out that the bottleneck is just the balancer. If this is stupidly one machine, then the service will fall 100%.

R
Roman Mirilaczvili, 2021-05-03
@2ord

My scheme is:
Write data: Clients --> LB --> API --> MQ broker --> queue processors --> DBMS
That is, the API receives data from the client, sends MQ to the broker (RabbitMQ/Apache Kafka) and immediately responds with status 200/202.
APIs and queue handlers scale as needed. A record in a materialized data view. DBMS with master-slave replication.
Reading data: LB --> API --> cache/DBMS
Here you can even take some framework like RoR/Django.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question