How to make broken builds not interfere with colleagues?

Question

Java

There is a repository, a lot of people make commits to it. Projects are built, unit tests are launched, and if the build did not break down, and the unit tests were successful, there is still trouble - the database may become unusable after them and the correct unit tests will break next time. Then we write a script that creates a test environment from scratch on a clean machine, builds it, then runs the tests, if everything was successful, we make a commit, otherwise we don’t do it and send an email to the culprit. Everything seems to be fine, but no, this whole construction works for a long time, what if the second commit has arrived until the first one has been completed - you can create a second VM and work out the second commit there. But what if the commits themselves work fine, but when they are together, they break the build? Is there some correct theory and correct tools that allow everyone to work together on a project without the risk of crashing builds?

Answer 1

Offhand, several problems are immediately visible.
First, no tests should change the base after themselves. If these are integration or acceptance tests, then, for example, they can be carried out in a transaction that is rolled back after the completion of each test (in addition, you can use in-memory DB - this will also speed up the execution of tests). And if these are unit tests, then they should not go into the database at all - you need to rewrite the code in such a way that unit testing does not require calls to the database (this, by the way, as a rule, improves the overall quality of the code and makes the code easier to reuse).
Secondly, when tests are performed for hours, this is a sure symptom that something is wrong in the conservatory. Try to unlink the tests from the database - this should greatly speed them up. If it doesn’t help, divide the tests into categories and run only unit tests for each commit, and run integration and acceptance tests every two to three hours (depending on the duration of their execution).
In addition, if all builds are executed on the same machine, then they must be executed sequentially, not in parallel - this reduces overhead and overall build / test time. But if you run builds on different machines (for example, TeamCity can do this) - then yes, you can run it in parallel so that the computing resource does not stand idle :-)

Answer 2

In a good way, in such circumstances, one should move to a distributed version control system (that is, to DVCS). Well, for example,switch to Git. And then each developer on the machine will have his own repository, and not just one main (central, main) repository for uploading all edits there. Moreover, this is not enough: it is imperative, by all means, to supplement all this with a distributed testing system, that is, arrange things in such a way that each developer on his machine can personally check whether the code from his own repository is operable - and only then upload it to the main (central) repository. And again, you should not upload immediately, but first download fresh edits from the central repository, apply them (eliminating merge conflicts if necessary, if the central edits change the same piece of code as the developer’s own edits), then suddenly test for performance - and only then pour.

Answer 3

I think that running two VMs for different commits and merging both is the wrong solution. Why not run builds one by one?

Answer 4

The correct theory is to separate the work of different people/teams.
Everyone works in their own branch, then merges the changes from the master into it and submits it for testing, and after successful testing, it can become the master. The second commit should fall into another branch, from the first or from the master, and then everything in a circle, regardless of the fate of the first branch, unless, of course, it has become the master and the changes in it will not be transferred to the second.
At the same time, everything in the master is confirmed, reviewed and tested. Everyone does what he wants in his branch and as much as he wants, until he solves the problem and does not interfere with anyone.
And the fact that testing is done for a long time is a reason to organize the work so that it would be done automatically after hours. If the developer just wants to check his build, then it is worth dividing the tests into quick and complete ones. Actually, tests are the same story as with changes - they can also change, be added for the branch, until they fall into the master along with the corresponding changes.

Answer 5

Hmm... if a commit is related to a small fix, then it makes no sense to run absolutely all the tests - and in a large project it can take several days. Secondly, large features are not committed that often, so the case of crossing commits is clearly out of the ordinary - after all, the second commit can either wait until the first feature is tested, or commit both and test both at once. Thirdly, any change in a more or less large project is tested by the developer himself before the commit, developed in a separate branch and mandatory code review, so it’s very, very difficult to “accidentally” break something - all such accidents will fall off even before commit. The conclusion is one and very simple: "measure seven times - then cut off."
Yes, I also heard such a practice - who breaks the build - puts pizza on the whole office. They say it helps a lot better than any unit tests ;)

Answer 6

On my projects, I made it so that testing starts each time on a new JVM and with Embedded DB. Instead of JBoss, I began to use Embedded JBoss, which runs inside this JVM, for the Web layer I use Jetty. I replaced all connections to external systems with stubs that work strictly according to the specifications of these systems, I connected some stubs directly, without using sockets and JMS.
The most important thing was to change the ports opened by JBoss and Jetty at startup so that you can start testing several projects at the same time.
As a result, about 550 tests on the most "heavy" project are completed on our test server in about 25-30 minutes (the machine is not very fast).
Other projects are being completed even faster.

Answer 7

To prevent the database from breaking, you need to use transactional tests - using spring, for example. In addition, you can use a test base, the dump of which is stored and changed in each branch separately.