How to ci/cd multiple services?

Question

Software Deployment

Greetings, I'm not sure if this is development or administration, but so be it. The Internet is full of articles on how to set up ci cd using one service as an example. But it is not entirely clear how to do this with a group of services.

The questions are:
1) Where and in what form is the cluster config stored
2) Monorep/multirep
3) Artifact versioning
4) How rollback occurs
5) How to update 2+ services at the same time.

What thoughts

I don't have a lot of Services, the main one and 3 satellites, while everything is in different turnips.

1) Config
If this is a monorep, then the config is stored in it. But, in what form are the versions of docker images written in it.

1. CI / CD writes the latest compiled version 1.1.111 there, after which the config is sent to the cluster
   
   • Pros:
     
     • directly from the config you can see which version of the image should be deployed (but is it necessary)
       
     
     
     
   • Minuses:
     
     • hard to freeze package version, ci/cd is always eager to update it
       
       
     
     
     
   
   
   
2. the config is static, manually set the version number, type: latest
   
   • Pros:
     
     • It is easy to freeze the version of the package, for production, the config with the latest tags, for tests in the feat-25 branch, we have the version feat-25>
       
     • To roll back from 1.1.111 to 1.1.107, delete the 1.1.111 image from the docker repository and set the latest tag to 1.1.107
       
     
     
     
   • Cons: it may not be clear from the config which version should be in operation (is it necessary?)
     There is a feeling that keeping a static config and managing service versions through image tags is more convenient and correct.
     

2) Mono-rep/multi -rep Mono-
rep
Usually, a huge repository is recorded in the minus mono-rep, in which a thousand people work, it is cloned for half a day.

This is not my case, the turnips are small, I work with them alone

. All services dependent on it were updated.
gitlab cicd will allow you to determine in pipelines which files in which directories have changed and, based on this, rebuild only the necessary packages. The cluster config is also here.

Multirep
- Seems harder to manage when linking to a cluster.
- The cluster config will be in a separate turnip deploy_cluster

3) Versioning.
Versioning each version by hand is sad, it is necessary that ci / cd do this. While the idea is this, hang up git tag major.minor occasionally with your hands and calculate the patch on ci / cd.

tag=$(git describe --tags --abbrev=0)
patch$(git rev-list ${tag}..HEAD --count)

Do I need to hang it on the git turnip using ci / cd (it is necessary to slightly correct the patch calculation algorithm), or use it only for versioning

images 1.1.107, some images may simply not have the real version 1.1.107

4) Rollback
It seems to be partially covered in the paragraph about config

5) How to update 2+ services at the same time.
This is necessary in order not to roll out service A that wants a new feature from service B, and service B has not yet been rolled out.

Well, there seems to be no magic here, you just need to roll out B first, and then A. And even if A rolls out earlier, it simply should not rise, and accordingly the cluster will not switch traffic to it.

Bonus question where to store env variables in docker stack/swarm. It has such concepts as config and secret, but they are mounted into the image as files, which is not always convenient.

What do adult devops have to say?

Answer 1

Each service has its own repo, its own jobs, its own deploys, often unrelated to each other.

Monorep / multirep

This is a question for developers, whichever is more convenient for you. Monorep is usually used when your services actually have a common code. If not, then it's easier in separate repositories.
Many CI/CDs can be configured to push to certain subdirectories, and then different services in the monorep will trigger their own jobs (teamcity for example)

Versioning

At the discretion of the developers. Maven has plugins for automatic semantic versioning. Or you can use your hands. The simplest thing is to put down the major version with your hands, and ci just adds the build number and / or the commit hash to the version.

Rollback

In the case of a cuber / openshift, there are healthcheck samples for this task, which you configure and that's it. If the application did not pass the healthcheck when it was raised, then the defield fails and that's it - the previous version will automatically continue to work. In the case of databases and other things, developers must write scripts for rolling forward / rolling back, or if the database is small, you can make backups or snapshots. It depends on the architecture

How to update 2+ services at the same time.

All right. Either you have the order of deploying services pre-specified in the processes, or half-hand output. But for good, such incompatibilities should not lead to a crash of the service. Well, feature B will not work temporarily in the service, because another one is not available, and that's it. This other service will rise and everything will work. This is the correct and ideal option that does not interfere with deployment. Deployment is carried out during non-business hours and does not affect anyone. At a minimum, such situations should not occur so often, so it is quite possible to find a period of time for such non-standard deployments.

It has such concepts as config and secret, but they are mounted into the image as files, which is not always convenient.

Both configs and secrets can be mounted both as files and as environment variables. Read the documentation.