How to organize two active Internet channels, FreeBSD 11?

Question

FreeBSD

Good afternoon! There was a need to deploy a server ( FreeBSD 11 ) with two active Internet channels. One channel is a local provider (ppp connection, tun0 ), the other is an Intertelecom 3G modem ( tun1 ). Both channels must be configured to forward ports to a computer on the local network. The IP addresses of both providers are dedicated.
The problem arises that on the second channel the redirection does not work. If the channels are "swapped" - the situation repeats itself: the one that became the first (Intertelecom) redirects, which became the second - no. I re-read a bunch of information, forums, the guys get it - I don't. The more I read on this topic, the more porridge in my head ... What have I missed or what am I doing wrong? XXX.XXX.XXX.XX1
Description of addresses:
- Provider 1 gateway IP address
XXX.XXX.XXX.XX2 - Provider 1 dedicated IP address
YYY.YYY.YYY.YY1 - Provider 2 gateway IP address
YYY.YYY.YYY.YY2 - Provider 1 dedicated IP address
192.168 .0.100 - IP address of the computer on the local network to which you want to configure redirection
. The kernel is compiled with the following options:

options         IPFIREWALL                     
options         IPFIREWALL_DEFAULT_TO_ACCEPT    
options         IPFIREWALL_VERBOSE              
options         IPFIREWALL_VERBOSE_LIMIT=50     
options         IPDIVERT                       
options         IPFIREWALL_NAT                  
options         LIBALIAS                    
options         ROUTETABLES=2
options         DUMMYNET
options         HZ="1000"

rc.conf settings :

ifconfig_re1="inet 192.168.0.2 netmask 255.255.255.0"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

gateway_enable="YES"            # разрешаем пересылку пакетов между се
firewall_nat_enable="YES"       # включаем ядерный nat
firewall_nat_interface="re0"    # указываем внешний интерфейс для nat
firewall_enable="YES"           # включаем ipfw
firewall_script="/usr/local/etc/ipfw/rc.firewall" # указываем путь, где будут

usbd_enable="YES"

ppp_enable="YES"
ppp_mode="ddial"
ppp_nat="YES"
ppp_profile="ISP1"
ppp_user="root"

The tun1 connection is raised with the command:
ppp -ddial -unit 1 ISP2
ifconfig:

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 98:de:d0:03:11:57
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 98:de:d0:03:f4:e1
        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
re2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 38:d5:47:a9:84:52
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (none)
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        options=80000<LINKSTATE>
        inet XXX.XXX.XXX.XX2 --> XXX.XXX.XXX.XX1  netmask 0xffffffff
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun
        Opened by PID 581
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet YYY.YYY.YYY.YY2 --> YYY.YYY.YYY.YY1  netmask 0xffffffff
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun
        Opened by PID 1168

ipfw rules :

# XXX.XXX.XXX.XX1 - IP-адрес шлюза провайдера 1
# XXX.XXX.XXX.XX2 - Выделенный IP-адрес провайдера 1
# YYY.YYY.YYY.YY1 - IP-адрес шлюза провайдера 2
# YYY.YYY.YYY.YY2 - Выделенный IP-адрес провайдера 1
# 192.168.0.100 - IP-адрес компьютера в локальной сети, на который необходимо настроить перенаправление

cmd="ipfw -q"

$cmd -f flush

$cmd add 100 allow ip from any to any via lo0

# Запрещаем доступ из вне к нутреннему интерфейсу
$cmd add 200 deny ip from any to 127.0.0.0/8
$cmd add 300 deny ip from 127.0.0.0/8 to any


# Разрешаем подключение по ssh
$cmd add 400 allow tcp from any to XXX.XXX.XXX.XX2 22 in via tun0
$cmd add 410 allow tcp from XXX.XXX.XXX.XX2 22 to any out via tun0 established

$cmd add 420 allow tcp from any to YYY.YYY.YYY.YY2 22 in via tun1
$cmd add 430 allow tcp from YYY.YYY.YYY.YY2 22 to any out via tun1 established

# Разрешаем DNS запросы
$cmd add 500 allow udp from any to XXX.XXX.XXX.XX2 53 in via tun0
$cmd add 510 allow udp from XXX.XXX.XXX.XX2 53 to any out via tun0 established

$cmd add 520 allow udp from any to YYY.YYY.YYY.YY2 53 in via tun1
$cmd add 530 allow udp from YYY.YYY.YYY.YY2 53 to any out via tun1 established


# Эти правила должны обеспечить активность двух каналов 
$cmd add 600 fwd XXX.XXX.XXX.XX1 ip from XXX.XXX.XXX.XX2 to any
$cmd add 610 fwd YYY.YYY.YYY.YY1 ip from YYY.YYY.YYY.YY2 to any


# Разрешаем некоторые типы ICMP траффика - эхо-запрос, эхо-ответ и время жизни пакета истекло
$cmd add 700  allow icmp from any to any icmptypes 0,8,11


# Разрешаем все соединения на локальном интерфейсе
$cmd add 800 allow all from any to any via re1


# Первый NAT
$cmd nat 1 config log if tun0 reset same_ports deny_in redirect_port tcp 192.168.0.100:3389 3389
$cmd add 1100 nat 1 ip from any to any via tun0

# Второй NAT
$cmd nat 2 config log if tun1 reset same_ports deny_in redirect_port tcp 192.168.0.100:3389 3389
$cmd add 1200 nat 2 ip from any to any via tun1


# Разрешаем все остальные
$cmd add 65535 allow ip from any to any

sysctl.conf:
net.inet.ip.fw.one_pass=0
setfib 0 netstat -rn:

Internet:
Destination        Gateway            Flags     Netif Expire
default            XXX.XXX.XXX.XX1    UGS        tun0
XXX.XXX.XXX.XX1    link#5             UHS        tun0
127.0.0.1          link#4             UH          lo0
YYY.YYY.YYY.YY1    link#6             UHS        tun1
192.168.0.0/24     link#2             U           re1
192.168.0.2        link#2             UHS         lo0
XXX.XXX.XXX.XX2    link#5             UHS         lo0
YYY.YYY.YYY.YY2    link#6             UHS         lo0

setfib 1 netstat -rn:

Internet:
Destination        Gateway            Flags     Netif Expire
default            YYY.YYY.YYY.YY1    UGS        tun1
XXX.XXX.XXX.XX1    link#5             UH         tun0
127.0.0.1          link#4             UH          lo0
YYY.YYY.YYY.YY1    link#6             UH         tun1
192.168.0.0/24     link#2             U           re1

ipfw show:

00100    0      0 allow ip from any to any via lo0
00200    0      0 deny ip from any to 127.0.0.0/8
00300    0      0 deny ip from 127.0.0.0/8 to any
00400  207  29610 allow tcp from any to XXX.XXX.XXX.XX2 dst-port 22 in via tun0
00410  151  47696 allow tcp from XXX.XXX.XXX.XX2 22 to any out via tun0 established
00420   23   3278 allow tcp from any to YYY.YYY.YYY.YY2 dst-port 22 in via tun1
00430   14   6180 allow tcp from YYY.YYY.YYY.YY2 22 to any out via tun1 established
00500    0      0 allow udp from any to XXX.XXX.XXX.XX2 dst-port 53 in via tun0
00510    0      0 allow udp from XXX.XXX.XXX.XX2 53 to any out via tun0 established
00520    0      0 allow udp from any to YYY.YYY.YYY.YY2 dst-port 53 in via tun1
00530    0      0 allow udp from YYY.YYY.YYY.YY2 53 to any out via tun1 established
00600  108   7484 fwd XXX.XXX.XXX.XX1 ip from XXX.XXX.XXX.XX2 to any
00610   28   1596 fwd YYY.YYY.YYY.YY1 ip from YYY.YYY.YYY.YY2 to any
00700   20   1120 allow icmp from any to any icmptypes 0,8,11
00800 2846 708794 allow ip from any to any via re1
01100 1305 445563 nat 1 ip from any to any via tun0
01200   16   1159 nat 2 ip from any to any via tun1
65535 2155 835010 allow ip from any to any

One gets the impression that the packets that arrive on the second channel leave through the first. How can this problem be solved using IPFW ?
Thanks in advance!

Answer 1

The IPDIVERT option in the kernel is superfluous, it is not needed for kernel NAT to work. It does not interfere with it, but just in case, keep in mind - it can be safely removed.
The first fwd (rule 600) is also not needed, and so traffic will fly to the default gateway without this rule.
And it doesn’t work for you because it never comes to the second NAT. The scheme is this:
In your case, the easiest way would be to give host 192.168.0.100 another IP address (say, 192.168.0.101), and wrap packets to it through provider 2. Then it will be possible to do this:

$cmd nat 1 config log if tun0 reset same_ports deny_in redirect_port tcp 192.168.0.100:3389 3389
$cmd add 1100 nat 1 ip from 192.168.0.100 to any via tun0
# Second NAT
$cmd nat 2 config log if tun1 same reset_ports deny_in redirect_port tcp 192.168.0.101:3389 3389
$cmd add 1200 nat 2 ip from 192.168.0.101 to any via tun1
$cmd add 1300 fwd YYY.YYY.YYY.YY1 ip from YYY.YYY.YYY.YY2 to any out via tun0

The rules are sketchy, and do not take into account that you need a regular nat for all other clients. Just a demonstration to understand the logic.

Answer 2

The correct impression is created - go to the default gateway. And this is normal behavior.
Although FreeBSD supports multiple gateways by default, it does not work the way you want (as far as I remember, it changes these gateways from package to package by round-robin, this is clearly not what is needed).
You can do this with nat global and multiple routing tables.
You can use ipfw probe to balance outgoing packets.
Read this article: ipfw.ism.kiev.ua/pbr.html, although it is old and natd is used there, the same can be done on nuclear nat.
PS: I implemented this task 3 years ago, it took a lot of time to debug. True, I had a more complicated configuration and the option from the article did not help me much.

Answer 3

When sending, the gateway must also be specified (setfib 0-1 )