Control and management of a large pool of web services

Question

Monitoring

Initial data: there is a system consisting of a web front-end and a large number of back-end web services that perform various functions, for example:

* Service for issuing photos
* Service for issuing video
* Service for converting video
* Service for servicing business logic related to users
* Service for accessing an internal database
* Service for accessing an external database
*…

To understand the essence of the issue, let's exaggerate the problem a bit and imagine that there are a very large number of services: 100, 200.

Each service has a network address and, for simplicity, is a JSON-Rpc service. Also, for simplicity, we will assume that all these services inherit some basic functionality, for example, they all know how to issue their version, turn off on an external command (if such functionality is needed), etc.

Task: to control the start, shutdown, monitoring of these processes. In particular, since many of these services are often physically located on the same server - it would be convenient at least to have scripts to start all processes at once or shut them down. Also, in order to understand if all the service configuration is working, it is useful to poll the services after running these scripts to make sure that all are started or all are turned off.

Goals: ease of deployment, monitoring, diagnostics of a system consisting of a large number of processes, possibly located on different servers.

Question: Are there any known solutions, possibly useful ready-made utilities for this kind of tasks, concepts and / or practices, and what practical advice can there be?

Note: When answering this question, you do not need to take into account the specifics of any specific servers or service hosting services. In principle, it can be conventionally assumed that the web server, services and all system components are created from scratch and therefore configuration schemes, service and / or service behavior, deployment mechanism and other aspects can be changed as you like to achieve the desired convenience.

Answer 1

Firstly, virtualization, no 10 services on one server, each service in its own virtualization container. to manage all virtual servers, something like solus or a second proxmox.
Deployment via Hudson
Monitoring zabbix Server
management via cf and puppet

Answer 2

> Goals: ease of deployment, monitoring, diagnostics of a system consisting of a large number of processes, possibly located on different servers.
ssh + authorization by keys. You will not find a ready-made solution for your specific task. You can also look at all sorts of monitoring systems, cfengine. Depending on the distribution, create a local repository with rpm/deb and deploy things via packages.

Answer 3

exactly deployment - you can look in the direction of systemimager va\\plus tying the configuration through svn