How to combine dev and stage environment for tests?

Question

Software testing

The problem is the following:

On a large web project, there is a development environment (dev) that looks into databases filled with a slice of data from a live environment.
There is a second development environment (stage) that looks into live databases, caches, etc.

Attention to the question:
Under what environment should I write tests?

In the dev environment, the data is already quite old, it is very difficult to drag and drop regularly from a live one, you need to carry terabytes of data to hundreds of different databases. In addition, due to the scale, the test environment is damn difficult to maintain: quite often something will fall off somewhere.

Everything is more or less good in the stage environment. The data is up-to-date and easy to work with. But there is one important BUT: developers have access only to the dev environment (for security reasons). To lay out a project in stage, you need to run a special spreader that works within a minute. That is, you have to commit, run the spreader and see your code looking at the live one. This is extremely tiring.

I want to write tests, but under what environment? For dev, there is often not enough data relevance (the best data is up-to-date), in order to run tests on the stage, you need to commit, decompose, etc. - tedious.

How do you solve the problem? Suggest a solution for this situation?

Answer 1

Are you sure that you need a live database (slice, etc.) to run the tests?
The generally accepted approach is that the test base is a separate base (and the environment is separate) for which data is filled using generators (factory_girl for example). In tests, each individual test is wrapped in a transaction with a rollback at the end for automatic cleanup. At the same time, reference tables are often filled in advance.
If a bug occurs that manifests itself with a certain set of data, then it is reproduced in tests with the necessary data.

Answer 2

As one of the options (not suitable in all cases) - from time to time do a lvm snapshot of the main base for developers. You can even take a snapshot of the replicated copy.

Answer 3

As an option, create a cron that every X minutes will have a spreader, and after its work is over, run the tests. And send yourself an email for failed tests. Those. we committed, we continue to work, after a while the mail arrives, we look at what is there (or then we look when there is time).

Answer 4

I would recommend setting up a feng shui dev environment. And then it will be difficult to cover semi-working functionality with tests.
I remember when one of the databases in one of the daemons was transferred from m to t, in order to work with data on a test environment with the necessary database, I had to wait for it to overflow from a live one, instead of getting a link with a working code that would allow me to fill in the data for the tests myself or call a test script - which would pour in that way a hundred new pseudo-random records automatically, and at the same time would check the normal operation of creating new records in the database in a test environment.