Why doesn't ip rule add fwmark work?

Question

PHP

I set up several uplinks on the gateway.
The gateway also runs openvpn inside the lxc container (the network is organized through a bridge interface, which includes lxc network interfaces and physical eth0 looking at the LAN)

[email protected]:/etc/init.d# brctl show
bridge name	bridge id		STP enabled	interfaces
br-eth0		8000.002191ef8b35	no		eth0
              vethGPYWS8
              vethXGQ5EQ
lxcbr0		8000.000000000000	no

Communication via udp/tcp protocols with services running on the gateway works fine for clients from the Internet.
It is not possible to configure forwarding of the OPENVPN server operating via udp.
The bottom line is that as soon as I add the rule
ip rule add from 192.168.128.13 lookup isc-ml
Where 192.168.128.13 is the ip of the openvpn container running on the gateway,
then the client normally establishes a connection to the server.
And the rule needs to work:

ip rule add from all fwmark 0x2 lookup isc-ml prio 1000

Additional Information:

[email protected]:/home/sysadmin# ip rule list
0:	from all lookup local 
1000:	from all fwmark 0x2 lookup isc-ml 
32759:	from all to 95.X.X.0/24 lookup isc-ml 
32760:	from 95.X.X.40 lookup isc-ml 
32762:	from all to 83.Y.Y.194 lookup isc-rt 
32763:	from 93.Y.Y.67 lookup isc-rt 
32766:	from all lookup main 
32767:	from all lookup default

Trace iptables
Where - 93.ZZ117 - ip openvpn client

Aug 14 13:01:30 gate kernel: [934528.016392] TRACE: mangle:PREROUTING:rule:2 IN=ethmiddle OUT= MAC=90:94:e4:82:0e:00:68:05:ca:0f:20:aa:08:00 SRC=93.Z.Z.117 DST=95.X.X.40 LEN=42 TOS=0x00 PREC=0x00 TTL=53 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 MARK=0x2 
Aug 14 13:01:30 gate kernel: [934528.016425] TRACE: mangle:PREROUTING:rule:4 IN=ethmiddle OUT= MAC=90:94:e4:82:0e:00:68:05:ca:0f:20:aa:08:00 SRC=93.Z.Z.117 DST=95.X.X.40 LEN=42 TOS=0x00 PREC=0x00 TTL=53 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 
Aug 14 13:01:30 gate kernel: [934528.016498] TRACE: nat:PREROUTING:rule:6 IN=ethmiddle OUT= MAC=90:94:e4:82:0e:00:68:05:ca:0f:20:aa:08:00 SRC=93.Z.Z.117 DST=95.X.X.40 LEN=42 TOS=0x00 PREC=0x00 TTL=53 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 MARK=0x2 
Aug 14 13:01:30 gate kernel: [934528.016557] TRACE: mangle:FORWARD:policy:1 IN=ethmiddle OUT=ethmiddle MAC=90:94:e4:82:0e:00:68:05:ca:0f:20:aa:08:00 SRC=93.Z.Z.117 DST=192.168.128.13 LEN=42 TOS=0x00 PREC=0x00 TTL=52 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 MARK=0x2 
Aug 14 13:01:30 gate kernel: [934528.016600] TRACE: filter:FORWARD:policy:12 IN=ethmiddle OUT=ethmiddle MAC=90:94:e4:82:0e:00:68:05:ca:0f:20:aa:08:00 SRC=93.Z.Z.117 DST=192.168.128.13 LEN=42 TOS=0x00 PREC=0x00 TTL=52 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 MARK=0x2 
Aug 14 13:01:30 gate kernel: [934528.016633] TRACE: mangle:POSTROUTING:rule:1 IN= OUT=ethmiddle SRC=93.Z.Z.117 DST=192.168.128.13 LEN=42 TOS=0x00 PREC=0x00 TTL=52 ID=8720 DF PROTO=UDP SPT=44619 DPT=1194 LEN=22 MARK=0x2

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i ethmiddle      -j MARK --set-mark 0x2
iptables -t mangle -A POSTROUTING -o ethmiddle     -j MARK --set-mark 0x2
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

Probros

iptables -t nat -A PREROUTING -d $XINETIP -p UDP --dport 1194 -j DNAT --to-destination $VPNSERVER:1194

Why doesn't ip rule fwmark work, but ip rule add from vpnserver does?
UPDATE 1:
Tcpdump on gateway.
with fwmark not working:

00:00:07.106369 IP (tos 0x0, ttl 53, id 16663, offset 0, flags [DF], proto UDP (17), length 42)
    93.Z.Z.117.43148 > 95.X.X.40.openvpn: [udp sum ok] UDP, length 14
00:00:00.000056 IP (tos 0x0, ttl 52, id 16663, offset 0, flags [DF], proto UDP (17), length 42)
    95.X.X.40.43148 > openvpn.cvision.lab.openvpn: [udp sum ok] UDP, length 14

with the added rule ip rule add from 192.168.128.13 lookup isc-ml - the forwarding works and the connection is established:

00:00:00.000000 IP (tos 0x0, ttl 53, id 41756, offset 0, flags [DF], proto UDP (17), length 42)
    93.Z.Z.117.57380 > 95.X.X.40.openvpn: [udp sum ok] UDP, length 14
00:00:00.000832 IP (tos 0x0, ttl 63, id 43790, offset 0, flags [DF], proto UDP (17), length 54)
    95.X.X.40.openvpn > 93.Z.Z.117.57380: [udp sum ok] UDP, length 26
00:00:00.044444 IP (tos 0x0, ttl 54, id 41767, offset 0, flags [DF], proto UDP (17), length 50)
    93.Z.Z.117.57380 > 95.X.X.40.openvpn: [udp sum ok] UDP, length 22
00:00:00.000089 IP (tos 0x0, ttl 54, id 41768, offset 0, flags [DF], proto UDP (17), length 142)
    93.Z.Z.117.57380 > 95.X.X.40.openvpn: [udp sum ok] UDP, length 114
...

By the way, I don't understand why I see this

95.X.X.40.43148 > openvpn.cvision.lab.openvpn: [udp sum ok] UDP, length 14

It turns out that in some strange way the gateway itself initiates a connection to the internal server from its external interface?
Perhaps this was due to SNAT without a source. Now I have added a LAN to the SNAT rule, and in theory this will no longer be the case. But these dumps were apparently made earlier, when SNAT was not so accurate. Previously, with one provider, it worked fine. Accidentally found in the examples of the people -s $LANNETIP

Answer 1

if there is authorization - generally easy, if not - store the ip from which the user comes in, this of course will not be an accurate metric, but + - will make it clear how many uniques there are.

Answer 2

Look at the IP address. Also read about HTTP ETag, in case it is supported by the application.

Answer 3

SOLUTION SEE. BELOW. Thanks to everyone who tried to figure it out.
There is

[email protected]:/etc/init.d# ip ro sh ta isc-ml
default via 95.X.X.1 dev ethmiddle

The matter is that through fwmark does not work.
I'm also confused by the martians packages logs:

Aug 14 11:56:30 gate kernel: [930628.627386] IPv4: martian source 192.168.128.13 from 93.Z.Z.117, on dev ethmiddle
Aug 14 11:56:39 gate kernel: [930636.846542] IPv4: martian source 192.168.128.13 from 93.Z.Z.117, on dev ethmiddle
Aug 14 11:56:54 gate kernel: [930652.586677] IPv4: martian source 192.168.128.13 from 93.Z.Z.117, on dev ethmiddle
Aug 14 11:57:08 gate kernel: [930665.944650] ll header: 00000000: ff ff ff ff ff ff 4c b1 6c 42 d1 7c 08 06        ......L.lB.|..
Aug 14 11:57:22 gate kernel: [930680.503274] ll header: 00000000: ff ff ff ff ff ff 4c b1 6c 42 d1 7c 08 06        ......L.lB.|..
Aug 14 11:57:26 gate kernel: [930684.060642] IPv4: martian source 192.168.128.13 from 93.Z.Z.117, on dev ethmiddle
Aug 14 11:57:32 gate kernel: [930690.082268] ll header: 00000000: ff ff ff ff ff ff 4c b1 6c 42 d1 7c 08 06        ......L.lB.|..
Aug 14 11:57:32 gate kernel: [930690.813437] IPv4: martian source 192.168.128.13 from 93.Z.Z.117, on dev ethmiddle
Aug 14 11:58:04 gate kernel: [930721.847001] ll header: 00000000: ff ff ff ff ff ff 00 25 22 89 80 df 08 00        .......%".....

When there is no ip rule add from 192.168.128.13 lookup isc-ml
I disabled rp_filter but that didn't work. If someone can tell me why in my situation there are such occasions - I will be grateful.
-----
SOLUTION:
OPENVPN will not work, more precisely CONNMARK is not suitable. The UDP protocol does not use connections. But I already knew about this. Just stupid.
Understood the operation of CONNMARK for the TCP protocol (using FTP as an example). Business was in the order of following according to rules of routing. By default, on linux, the default route is placed in the main table (in our case, this is done by the daemon for pppoe from rt). The routing rule added to the beginning of the rule chain, depending on mark, worked only for incoming packets that were sent immediately back.
Something like this:
Strange isn't it?
And in the main table, the default route sent all packets from the FTP server through the main provider.
The output is simple use iptables log like this
-I - adds a rule to the beginning of the chain - this is important, otherwise if ACCEPT or DROP work, then the packet will not reach the LOG rule and you will not see anything in the syslog.
In essence, you had to look into the routing tables and check each rule to see if it could send through the rt provider. It turned out it could. Now I hope to screw it all up as it should be in a Debian-based system and I will try very hard to write another article)

Answer 4

it looks like you don't have the required routes in the isc-ml table
ip ro sh ta isc-ml