How to configure ntp for ceph?

Question

linux

Greetings!
There is a task to set up a failover cluster of three nodes on Proxmox. As a data storage, it was decided to use Ceph on the same three nodes. For it, you need to set up time synchronization so that the difference between the nodes does not exceed 50 ms (this is the question of why I need such accuracy).
Internal network 192.168.0.0/22.
We have a separate Debian 9 server (vt-gate) with ntpd configured:

/etc/ntp.conf

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntpd.log
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
pool 0.ru.pool.ntp.org iburst
pool 1.ru.pool.ntp.org iburst
pool 2.ru.pool.ntp.org iburst
pool 3.ru.pool.ntp.org iburst
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1
restrict 0.ru.pool.ntp.org
restrict 1.ru.pool.ntp.org
restrict 2.ru.pool.ntp.org
restrict 3.ru.pool.ntp.org
restrict 192.168.0.0 mask 255.255.252.0 nomodify notrap

Since I do not have extra hardware, the test bench consists of three virtual machines running Proxmox 5 with the same Proxmox on board. Internal network addresses of nodes 192.168.3.11, 192.168.3.12, 192.168.3.13.

/etc/ntp.conf

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntpd.log
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server vt-gate.crb.local iburst prefer
#Для первой ноды
peer 192.168.3.12 iburst
peer 192.168.3.13 iburst
#Для второй ноды
#peer 192.168.3.11 iburst
#peer 192.168.3.13 iburst
#Для третьей ноды
#peer 192.168.3.11 iburst
#peer 192.168.3.12 iburst
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited
restrict 127.0.0.1
restrict ::1
restrict vt-gate.crb.local
#Для первой ноды
restrict 192.168.3.12
restrict 192.168.3.13
#Для второй ноды
#restrict 192.168.3.11
#restrict 192.168.3.13
#Для третьей ноды
#restrict 192.168.3.11
#restrict 192.168.3.12
restrict 192.168.0.0 mask 255.255.252.0 nomodify notrap

The idea is that all nodes are synchronized with the vt-gate, but if it is unavailable, the nodes are synchronized with each other so as not to get a difference of more than 50 ms.
So this doesn't work. Judging by ntpq -p, the difference can be up to seconds.
Tell me how to cook all this correctly?
Thank you.
PS For some unknown reason, the nodes do not write logs to /var/log/ntpd.log (no access), although ntp runs as root.
UDP 0:
I remembered from previous experiments that when synchronizing only with vt-gate, Ceph did not swear for a while, but as soon as one more synchronization source was added (another local server under Windows), a discrepancy immediately appeared.
UDP1:
I remembered about systemd-timesyncd. In theory, it should be turned off after installing ntp, but just in case, I performed the following manipulations to turn it off:
# cp /lib/systemd/system/pve-cluster.service /etc/systemd/system/pve-cluster.service
In the file / etc/systemd/system/pve-cluster.service changed
Wants=systemd-timesyncd.service
to
Wants=ntp.service
# systemctl daemon-reload
Checked systemctl status systemd-timesyncd says Active: inactive (dead).
So far, ntpq -p gives a discrepancy between nodes of no more than 27ms:

[email protected]:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+192.168.3.11    192.168.0.4      3 s  106  512  373    0.165    0.027   0.015
+192.168.3.12    192.168.0.4      3 s  446  512  367    0.174    0.022   0.033
*vt-gate.crb.loc 192.36.143.130   2 u  247  512  377    0.183   -0.003   0.053

Answer 1

Ceph is not so time-critical, we sometimes have a run-up of up to 5 minutes! And it's better to do this, run one additional ntp, and synchronize with it.
In other words, vt-gate -> ntp on one of the nodes -> other ntp + vt-gate
Just set the right priorities!!!