How to implement model data synchronization in Django?

Question

Django

Good afternoon.
There is a test version of the application.
Some work settings are located in the models and it is impossible to put them into the code,
because you need to configure them through the admin panel.
Question: How to transfer data to a production server?
- Through sql dumps, it is long and difficult with dependencies (only the settings need to be transferred, and the documents should remain in place, that is, delete_all - insert_all is not very suitable)
- There is not enough RAM to use unloading and loading fixtures - project workflows with more than 5 million documents and fields, roles, stages, access rights there are a lot)
There are ideas for adding a field like auto_now=True and unloading diff's from the date of the previous migration, packing them into json files (like fixtures) and creating rollback files (containing the same diff's, but with the data that was before applying the changes) .
Sequential roll forward of several such files will lead to merging of versions.
But this is still only an abstract idea for my bike, which I don’t really want to write.
We somehow managed at the database level with crutch procedures at the database link level,
but if the test database and the production database are very far from each other (the test database is hosted, the combat one is inside the local network and is closed from the outside), and crutches with weekly releases are tiring.
I'm sure I'm not the first to encounter this.
Has anyone solved this problem and how?

Answer 1

A solution for creating incremental data for transferring to a combat server and rolling it up there.
We hang post_delete signals on the necessary models to save the model - id to the model of remote objects.
Here is the code for creating files.
A little later I will post a link to an article with a more detailed description:
- Creating change files
- Rolling forward with creating rollback files
- Rolling back changes

# coding=utf-8

from datetime import datetime
from os import makedirs, listdir
import json

from django.core import serializers
from django.core.management.base import BaseCommand
from optparse import make_option
from django.db.models.loading import get_model
from django.conf import settings


class Command(BaseCommand):
    help = """
    Create difsettings from timestamp. 
    Usage: manage.py docflow_diffsettings_create '2014-01-01'
    """

    option_list = BaseCommand.option_list + (
        make_option('--from',
            action="store", 
            type="string",
            dest='from',
            help='timestamp: changed from argument to current DateTime'),
        make_option('--project_id',
            action="store", 
            type="int",
            dest='project_id',
            help='Docflow project number'),
    )

    def __init__(self, *vargs, **kwargs):

        super(Command, self).__init__(*vargs, **kwargs)
        self._datetime_format = '%Y-%m-%d %H:%M'

    def handle(self, *args, **options):

        date_to = datetime.now()
        date_from = datetime.strptime(options['from'], self._datetime_format)

        project_id = options["project_id"]
        project_folder = "%s/docflow/diffsettings/%s" % (settings.BASE_DIR, project_id,)
        folder_path = "%s/%s > %s" % (project_folder, 
                                    date_from.strftime(self._datetime_format),
                                    date_to.strftime(self._datetime_format))
        
        makedirs(folder_path)

        JSONSerializer = serializers.get_serializer("json")
        json_serializer = JSONSerializer()

        models_paths = set()

        for model_path in args:
            models_paths.add(model_path)

        # deleted objects
        models_paths.add('docflow.ObjectsDeleted')

        # create files
        for model_path in models_paths:

            application_name, model_name = model_path.split('.')
            model = get_model(application_name, model_name)
            objects_changed = model.objects.filter(d_change__range=(date_from, 
                                                                    date_to))
            
            # Обязательно наличие файла, даже если нет ни одной измененной модели
            # это необходимо для создания бэкапа при накате настроек 
            # (если по модели нет изменившихся, но есть удаленные объекты)
            # Создание файла отката настроек происходит 
            # внутри итерации перебота этих файлов
            with open("%s/%s.json" % (folder_path, model_path), "w") as out:
                json_serializer.serialize(objects_changed, stream=out, indent=4)

Answer 2

Those. need to synchronize the new version of the application settings with the old one in production? Yes, South will not help you here) Make a manage command like update_settings, in which you check the compliance of the data in the models and add changes.

Answer 3

It seems like the task should be typical:
there is a test, there is a production, you need to merge the settings from the test into prod.
Is there really no such mechanism available and supported by someone.
Actually, I think in this direction.
If no one tells me a ready-made solution, I'll do it.
Thank you.