continuous deployment. Not only forward through the git commit history, but also urgently back?

Question

PHP

Good evening, comrades!
My question is related to building the deployment process.
Let's say we have a web project that periodically rolls out of the master branch to the production server using jenkins. Of course, the database migrates and everyone is happy.
Everything would be fine, but then a global fakap comes, nothing works, everyone is in shock, admins are woken up in the middle of the night, programmers check their commits "if only not mine." The situation, I think, is familiar to everyone.
We want to roll out the application constantly to avoid "Tomorrow release, everyone be nervous!". With a small delay for tests and immediately into battle after the commit, and so hundreds of times a day. Just like github =)
Now the crux of the matter is - how do the big uncles do a rollback to the version back when an unsuccessful commit hits production? We want to quickly roll back and thoroughly deal with the error over a cup of coffee! We don't want to go through sssh to a server! We do not want to write our bike, the reliability of which in any rash situation is zero!
I have an idea that some kind of chat like slack is needed, where there will be a correspondence with the ci server and at the right time we send it "Fu•k! Deploy previous version! Now!" =)
What tools? Are there any problems with the database from migrations back and forth? Do migration engines save from childhood illnesses? If so, which ones?
Thanks, comrades!

Answer 1

To be honest, I did not quite understand the question :) I will try to answer the question "how to quickly rollback to previous versions in a small web project".
Assumption 1: In principle, the project fits on one physical machine.
Assumption 2: Your web project consists of a web server and a database (at least, only they are updated during deployment). There are no MQs standing on the side, any middle-var and other interaction with other systems.
I think the best solution is to do this:
To roll back the code, it is enough to make a parameterized build in Jenkins, and pass the hash of the revision to which you need to rollback to this task (I personally run the "hg update -r ${REV_HASH} --clean" shell script as the first step in such a build, but maybe you can through plug-ins in gue to configure). Docker is good, but it is designed for highly distributed systems, and in a small project it seems to me that this is an overkill.
With database rollback, the issue is more complicated. Firstly, because in any case, making a "copy" of any database is more or less a lock. Although some databases are able to take snapshots very quickly, for example mongoDB is able to make a backup without stopping and blocking read / write without loss of consistency (under certain conditions, of course). Secondly, the optimal solution will depend on the size of your database and the load on it.
For example, if you have MySQL with 100 megabytes, then mysqldump, which is launched by Jenkins before the start of the deployment of the new version, will be a very simple and reliable solution (of course, saving the hash of the current revision, so that later along with the code to roll back).
If you have a 20GB database, then you can raise a second server with a slave base next to it, and lock it when dumping it with the same mysqldump.
If you don't want to fool around with the second unit, and the system allows you to block the base for a short time, then you can do snapshots using lvm, for example. (see www.lullabot.com/blog/article/mysql-backups-using-...
From the text, I understand that the question "how do they do a quick rollback to previous versions in a large project" is still of interest. I so understand that at "big uncles" to change architecture of a DB is very expensive and labor-consuming process. Especially if the structure is not backwards compatible (the same recalculations of all indexes, huge amounts of information, waiting for all replicas to be updated, etc.). Regarding the deployment, as I understand it, they also do not have a very large choice: these are either virtual machines (with docker, or something similar), or an application platform (either their own, or any PaaS's). Well, and then all the same, there is only one way: new virtual machines / containers are turned on, records in the load balancer are updated, old virtual machines / containers are turned off.

Answer 2

Capistrano
www.techchattr.com/easy-php-deployment-with-capistrano