Why don't OSPF Hellos go through the GRE/IPSec VPN tunnel one way?

Question

Computer networks

Greetings.
There is the following config: several servers united in a "local network" by connecting each to each using GRE tunnels, or rather GRETAP tunnels, i.e. L2 (vpntapX), encrypted with IPSec, on each server, the tunnels are bridged (vpnbr0), so all servers seem to be included in one switch. If anyone has a better way to implement a full-mesh network over the internet, I'd be happy to hear it, but the current solution worked decently until now when a 4th server needed to be added.
Each server is responsible for its own subnet in which virtual machines “live”, Quagga (OSPF, Zebra) is used for routing tunnels, but OSPF Hello does not.
OS and software versions
Old servers

• Debian 7 (Proxmox 3) Linux pve01 2.6.32-30-pve #1 SMP Wed Jun 25 05:54:15 CEST 2014 x86_64 GNU/Linux
  
• quagga 0.99.22.4-1+wheezy1
  

New server

• Debian 8 (Proxmox 4) Linux pve03 4.4.35-2-pve #1 SMP Mon Jan 9 10:21:44 CET 2017 x86_64 GNU/Linux
  
• quagga 0.99.23.1-1+deb8u3
  

Network setup
Good IPs - where everything works

• 172.20.128.1
  
• 172.20.128.2
  
• 172.20.128.254
  

Problematic IP - where OSPF does not work

• 172.20.128.3
  

VPN tunnel:

ip address show vpntap1
8: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast master vpnbr0 state UNKNOWN group default qlen 1000
    link/ether 4a:6c:b9:b6:47:06 brd ff:ff:ff:ff:ff:ff

Bridge connecting tunnels:

brctl show vpnbr0
bridge name     bridge id               STP enabled     interfaces
vpnbr0          8000.fa8d12bcc999       no              vpntap1
                                                        vpntap2
                                                        vpntap3

ip address show vpnbr0
131: vpnbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN
    link/ether fa:8d:12:bc:c9:99 brd ff:ff:ff:ff:ff:ff
    inet 172.20.128.1/24 brd 172.20.128.255 scope global vpnbr0
    inet6 fe80::f88d:12ff:febc:c999/64 scope link
       valid_lft forever preferred_lft forever

Config example:

auto vpntap1
iface vpntap1 inet manual
        pre-up ip link add $IFACE type gretap local A.B.C.D  remote X.Z.Y.F key 123456
        post-up ip link set $IFACE mtu 1400 || true
        post-down ip link delete $IFACE
auto vpntap2
iface vpntap2 inet manual
        pre-up ip link add $IFACE type gretap local A.B.C.D remote Q.W.E.R key 123456
        post-up ip link set $IFACE mtu 1400 || true
        post-down ip link delete $IFACE
auto vpntap3
iface vpntap3 inet manual
        pre-up ip link add $IFACE type gretap local A.B.C.D remote A.B.Z.Y key 123456
        post-up ip link set $IFACE mtu 1400 || true
        post-down ip link delete $IFACE

auto vpnbr0
iface vpnbr0 inet static
        address  172.20.128.1
        netmask  255.255.255.0
        bridge_ports regex vpntap[1-9].*
# We don't want disabled links so STP off
        bridge_stp off
        bridge_fd 2
        bridge_maxwait 5
# Do not forward packets over "ports" we don't want broascast storms
        pre-up ebtables -A FORWARD --logical-in $IFACE --j DROP
        post-down ebtables -D FORWARD --logical-in $IFACE --j DROP
# Route all multicast to this interface
        post-up  ip route add 224.0.0.0/4 dev $IFACE
        pre-down ip route del 224.0.0.0/4 dev $IFACE
# Clamp MSS to PMTU
        post-up  iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $IFACE -j TCPMSS --clamp-mss-to-pmtu -m comment --comment "$IFACE"
        pre-down iptables -t mangle -D POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $IFACE -j TCPMSS --clamp-mss-to-pmtu -m comment --comment "$IFACE"
# Disable multicast snooping
        post-up echo 0 > /sys/devices/virtual/net/vpnbr0/bridge/multicast_snooping

Routes
On 172.20.128.1 we see networks added by Quagga.

ip ro
172.20.252.2 via 172.20.128.2 dev vpnbr0  proto zebra  metric 20
172.20.252.254 via 172.20.128.254 dev vpnbr0  proto zebra  metric 20
A.B.C.F/27 dev vmbr0  proto kernel  scope link  src A.B.C.D
172.20.128.0/24 dev vpnbr0  proto kernel  scope link  src 172.20.128.1
172.21.254.0/24 via 172.20.128.254 dev vpnbr0  proto zebra  metric 20
10.17.1.0/24 dev tun0  proto kernel  scope link  src 10.17.1.1
172.21.2.0/24 via 172.20.128.2 dev vpnbr0  proto zebra  metric 20
172.21.1.0/24 dev vmbr1  proto kernel  scope link  src 172.21.1.1
224.0.0.0/4 dev vpnbr0  scope link
default via A.B.C.Z dev vmbr0

On 172.20.128.3 only local routes.

ip ro
default via Q.W.E.F dev vmbr0
Q.W.E.Z/26 dev vmbr0  proto kernel  scope link  src Q.W.E.D
172.20.128.0/24 dev vpnbr0  proto kernel  scope link  src 172.20.128.3
172.21.3.0/24 dev vmbr1  proto kernel  scope link  src 172.21.3.1
224.0.0.0/4 dev vpnbr0  scope link

Diagnostic information, ospf, tcpdump
As you can see, the working servers do not see the new server at all, because the OSPF Hello does not reach them.

show ip ospf neighbor
    Neighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
172.20.252.2      1 Full/DR           36.509s 172.20.128.2    vpnbr0:172.20.128.1      0     0     0
172.20.252.254    1 Full/Backup       35.370s 172.20.128.254  vpnbr0:172.20.128.1      0     0     0

On 172.20.128.1 OSPF Hellos of all servers except the new one are visible.

tcpdump -ni vpnbr0 proto ospf                    
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpnbr0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:45:42.335682 IP 172.20.128.254 > 224.0.0.5: OSPFv2, Hello, length 52
18:45:43.468166 IP 172.20.128.1 > 224.0.0.5: OSPFv2, Hello, length 52
18:45:43.469742 IP 172.20.128.2 > 224.0.0.5: OSPFv2, Hello, length 52
18:45:52.336008 IP 172.20.128.254 > 224.0.0.5: OSPFv2, Hello, length 52
18:45:53.468557 IP 172.20.128.1 > 224.0.0.5: OSPFv2, Hello, length 52
18:45:53.470197 IP 172.20.128.2 > 224.0.0.5: OSPFv2, Hello, length 52

The new server sees all neighbors, but hangs in Init, apparently because no one sees it.

Neighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
172.20.252.1      1 Init/DROther      31.781s 172.20.128.1    vpnbr0:172.20.128.3      0     0     0
172.20.252.2      1 Init/DROther      31.782s 172.20.128.2    vpnbr0:172.20.128.3      0     0     0
172.20.252.254    1 Init/DROther      30.647s 172.20.128.254  vpnbr0:172.20.128.3      0     0     0

All OSPF Hellos reach the problematic 172.20.128.3

tcpdump  -ni vpnbr0 proto ospf                                           
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 
listening on vpnbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:48:02.342019 IP 172.20.128.254 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:03.473027 IP 172.20.128.1 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:03.473527 IP 172.20.128.2 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:05.589296 IP 172.20.128.3 > 224.0.0.5: OSPFv2, Hello, length 56
18:48:12.342449 IP 172.20.128.254 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:13.473298 IP 172.20.128.1 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:13.473852 IP 172.20.128.2 > 224.0.0.5: OSPFv2, Hello, length 52
18:48:15.589541 IP 172.20.128.3 > 224.0.0.5: OSPFv2, Hello, length 56

Continued in the comment, limit is 10 thousand characters :(

Answer 1

Respect for the situation described in detail :-) I didn’t fix everything, but I have to offer the following :-) Some providers (noted by Rostelecom, Online) incorrectly work out either cgnat or some kind of ALG on the way. Which causes SOME types of packets to be blocked in tunnels or when negotiating tunnels. According to this scheme, L2TP sessions are not established for me - a certain packet is simply blocked in the client's response to the server, and that's it.
So I'm all for what. Try changing the tunnel type with the problematic IP. Or try enabling encryption on this tunnel to hide the traffic. Maybe then everything will work.