S
S
severnik2020-05-07 23:35:10
Data synchronization
severnik, 2020-05-07 23:35:10

What is the best way to implement continuous synchronization of files and their permissions between two servers on Linux and Windows?

There are two servers remote from each other on Ubuntu 16.04 and Win2012r2.
Both are pulled into the same domain with AD, both have file balls with many different access rights through AD.
In the balls of the order of 3 million. files in 400 thousand. directories. Just over 3TB of data.
The whole thing must be continuously synchronized with minimal delays between servers at least in one direction (from Linux to Windows) and, preferably, free solutions.

Considered options:
Resilio Sync - does not know how to synchronize extended attributes (xattr, ACLs), there is Connect, but there is already a fee,
Syncthing - also does not know how to synchronize extended access rights,
rsync - everything is perfectly synchronized, but it takes a long time to build a directory tree,
lsyncd - cannot be synchronized with Windows, tried to mount a Windows share via CIFS, but there is no inotify and nothing happened,
osync - also failed to synchronize with a mounted Windows CIFS directory, gives

No inotifywait command found. Cannot monitor changes.

Unison - does not know how to synchronize extended rights. Although the two-way synchronization on the test with parameters is -repeat watch -fastcheck truefaster than all the others, but on large amounts of data I did not wait for the initial build to finish,
robocopy is a built-in utility in Windows, similar to rsync in Linux, and I stopped on it for now, because. there are built-in parameters for monitoring changes in the source ball.

I run commands for each ball in several processes (to speed up):
ROBOCOPY \\serv01\share \\serv02\share /MIR /SEC /Z /R:1 /W:5 /REG /MT:32 /MON:1 /SECFIX /NFL

/MON:1 - monitors source changes for 1 minute after synchronization ends and restarts the process.
The problem is the same as that of rsync - long tree building, it seems impossible to disable this, as in lsyncd or Unison. If anyone knows how to disable the initial building of the tree and immediately transfer the changes, please tell me.
The fact that without initial synchronization I know less reliably, I plan to completely run rsync or robocopy once a day.

Is there anything else that can synchronize extended attributes in the same way, without building the tree first?
Robocopy is able to somehow monitor file system changes in the Linux ball, under Linux I did not find anything that would also work with balls, only lsyncd and its derivatives that monitor inotify.

In the long term, demolish Windows and make lsyncd or zfs replication everywhere, but this is a matter for the future, now we need a solution in the current configuration.

Answer the question

In order to leave comments, you need to log in

4 answer(s)
P
pindschik, 2020-05-08
@pindschik

Out of personal interest, I will add a question to the community: this problem does not exist in the Windows + Windows bundle (via DFS replication), but are there any alternatives that NORMALLY work in a similar way to DFS? Those. forming a replication buffer on the fly, with differential data compression, tracking rights, alternative file streams, STABLE, and at the same time not giving up like Sync in front of a storage that is somewhat larger and more complex than a home photo archive?

A
Alexey Dmitriev, 2020-05-08
@SignFinder

Samba has some kind of DFS support .
Perhaps in vain they abandoned the idea with DFS?
I'm not sure, but perhaps Samba shares can be added to the Folder Targets Namespaces, or set up one-way DFSR with Windows shares on the Samba share.

D
Dimonchik, 2020-05-08
@dimonchik2013

cyberduck.io + sftp or whatever
but it's not lsync of course

A
Alexander, 2020-05-08
@UPSA

Why didn't Ceph look?
Shared drive for

Ubuntu 16.04 and Win2012r2.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question