How to speed up the rollout of changes in a large project (monolith)?

Question

Software Deployment

Essence of the question. Large php project, monolith. The business demands a faster rollout of changes to open beta testing (it's almost a prod - there are active customers, everyone is in combat).
Changes produce about 6-7 developers daily. There are assembly servers, a bunch of tests (unit + functional). DB is one.
Now the release process is generally very good. Every two weeks:
- each change is checked at the stage of incorporation into dev (team lead)
- all ready changes are checked by the development team on the stage server + bugs found are fixed immediately - all ready changes
are checked by business analysts on the stage server
- bugs are reported
- bugs are fixed
- everything is rechecked
- hurrah! rolling out!
well, sad news - all this is about 3-4 days.. :)
We tried to answer this question for ourselves, here are my thoughts:
1. Start splitting the entire system into microservices, thereby trying to get away from deploying all this software in one sitting.
Pluses on the face, the code does not intersect, you can deploy any of the components separately, scaling individual services, etc.
Everything is cool, but! provided that the database in this case is used from the monolith and is still alone. If this is not the case, then this only adds more questions and those difficulties than
it was before (internal interaction of mikruh, data synchronization, the complexity of working with this goodness without normal orchestration)
2. Stupidly switch to the dev branch in open beta testing and release very often (and live in hell)))).
Also, the option is generally tenacious, if you plant an individual person to extinguish daily fires.
Gentlemen, I would like to know how you got out of such situations in your project, and how you went to this.
Thank you!

Answer 1

Perhaps you have grown to a big project. So, without analytics, you can’t figure it out.
You need to have a clear idea of ​​how much time is spent on each step of your release process.
If there are no unambiguous metrics - cover. You don’t need how many minutes by eye, but clearly statistics for each task - how long it took to collect, test, how long it took to release, how much to check by the business, and so on and so forth. There are few metrics.
Then, based on the collected data, draw graphs, charts, analyze what is spent the most time on.
Then think about where you can optimize.
To analyze and optimize without numbers and pictures with diagrams is the way to nowhere. You can guess with optimization, you can not guess.
When all the data is on hand, you can optimize.
Most often it helps to:
- Exclude from processes or minimize the participation of people (CI, CD, autotests, etc.)
- Parallelize multiple monotonous processes (merging, running tests on tasks, running tests on a build, etc.
) how all the tasks necessary for the business got into it)

Answer 2

Have you automated assembly, deployment, testing procedures?
In my understanding, if you come to work the next morning and see the version deployed on the test server with a test pass report, then the time from dev to prod is reduced to 2 days (stabilize in the morning and get a stable release in the afternoon).
Proven in practice. We collected not once every two weeks, but every night. Basic checks took place at night, and the analytics functionality was checked only when the update was being prepared by the customer.
Important: not the resulting assembly should be rolled onto prod, but all assemblies in exactly the same sequence in which they were rolled onto the stage (first an unstable release, then fixes to a stable one). This is necessary to avoid errors during reassembly.

Answer 3

1. Each new functionality is the replacement of one logical block with another or the addition of a new one. Accordingly, blocks of the same type must have the same interface.
2. For quick automatic rollback in case of an error, before starting the work of a new block, all input values ​​and the state of the parameters changed by it should be remembered.
3. In case of an exception, we roll back and apply the old block without requiring any additional action on either side.