J
J
jar3b2018-02-16 12:00:27
Service Oriented Architecture
jar3b, 2018-02-16 12:00:27

How to properly handle disconnect in microservices?

Situation: there is a certain number of microservices, let's say 20, Docker (swarm mode) is used. They have connections (both spot and persistent) to other services (databases, MQ brokers, etc.).
Question: how to handle connection errors with other services?
Options:
1. Try to reconnect, it makes sense if the connection is permanent, if "on demand", then you can implement a counter of unsuccessful attempts and then take action. Here the problem for me is that the reconnect logic will need to be implemented separately for each service (Mongo, Elastic, Postgres, Nats, etc) and, possibly, on different languages.
2. Immediately exit, that is, shut down the container, without any reconnects, and docker must restart the service itself. Here I see the following problem: during a mass update (for example, the version of the DB image has been updated), there will be a lot of restarts, it takes time, and it would probably be faster to try to reconnect (p. 1). I suspect there may be other problems as well.
3. Another option.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexander, 2018-02-18
@jar3b

Option 2 is clear.
There are options for swarm mode (I don’t remember the name exactly) that allows you to update services not all at once, but in parts (for example, 5 replications, then you can update 2 at a time). Plus there is a kind of timeout for this case. Look at some of Arun Gupta's talks about swarm, he talks very well.
Also in favor of the 2nd option, the simplicity of the application. No need to fence counters and stuff. The only thing is, if the service has some kind of state, you can make 1 attempt to reconnect and then fall.
The advantage of this approach will be visibility in the degradation of some service. Conventionally, the base has fallen, and all services working with it will fall, and this will be noticeable. And if the reconnect is inside the service, then it will no longer be visual and it will be necessary to debug and climb into the logs.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question