How to test complex distributed systems?

Question

Java

How do you test complex distributed systems in automatic mode? For example, consisting of several components - Cassandra, ActiveMQ, a couple of self-written daemons. In reality, this is all spinning on 8-10 machines, it became necessary to run basic tests in automatic mode.

Unfortunately, it will not work to limit ourselves to tests in each individual component - there have already been cases when changes in one of the components broke other components, but the tests in them did not fall - the interface simply changed, and other components were forgotten or nuances were not taken into account.

Until now, testing was carried out in manual mode, with installation on a test cluster, but firstly, it's lazy, and secondly, you get tired of checking everything with your hands.

A self-written solution is trite: raise a virtual machine (for example, on VirtualBox), pull it from the outside via ssh and send commands. But maybe I do not know some obvious and simple solution?

Answer 1

Look, for example, Vagrant
habrahabr.ru/post/140714/

Answer 2

The purpose of integration testing is to ensure that the interaction of certain subsystems does not violate the established contract.
You can use mocking - imitate the reference responses of some service. For example Mockito
In general, it would be nice to have functional tests that cover business requirements and run them on a QA environment that is as similar to production as possible.