Nginx: proxy_protocol + regular connection + X-Real-IP header, how to make it work together?

Question

Nginx

There was a need to forward (NAT) TCP connections to Nginx already working at an external address through HAProxy. Of course, in this case, we start getting server IP from HAProxy.
After reading the documentation, it was decided to try using the PROXY Protocol (now supported by Amazon, etc.).
Added another listen XX proxy_protocol to server {}; and directed HAProxy to it.
realip_header had to be set equal to proxy_protocol (instead of the "normal" X-Real-IP) and here we start getting funny problems. Because connections are proxied through proxy_pass to the backend, the real address of the visitor is also transmitted to it through the X-Real-IP header. However, if we simply use $remote_addr in it as before, we get the correct address at the output only if the person went directly to Nginx, in the HAProxy-> Nginx (proxy_protocol) we get the HAProxy IP ... If we take $proxy_protocol_addr, then the person who came directly, will not have it at all.
Tried playing around with map and checking for proxy_protocol_addr, but it still doesn't work.
Maybe someone faced a similar problem? This is only a basic question so far - then you need to configure the bundle even worse:
HAProxy->Nginx (proxy_protocol)
Nginx->Nginx (regular)
Direct to Nginx
And in all cases, find the real address of the client...
If you need any configs, I can provide.
Thank you for your attention.
UPDATE: Decided by myself... I highlight the comment. If there is a better solution, I will gladly accept and update.

Answer 1

Here is the final solution, from the minuses:
1) Relies on CF-Connecting-IP (this is from Cloudlflare)
2) The access_log will log the IP of the external Nginx, although at the end the real client IP ($http_x_forwarded_for) will be visible, you can replace log_format
3) Bug feature: if you move all proxy_set_headers outside the virtual host (server {}), then they are NOT applied in proxy_pass. Since they are not processed in its context. Reason: map is a dynamic thing according to the documentation (Since variables are evaluated only when they are used, the mere declaration even of a large number of “map” variables does not add any extra costs to request processing) and it is only processed if " needed"... Why does Nginx consider it not needed if it is described in nginx.conf (and even correctly used in log_format), and not in server {}? Apparently there are reasons for that.
So, first, the general settings:
===== nginx.conf OR conf.d/000-realip.conf =====
set_real_ip_from 1.1.1.1/24; # haproxies internal
set_real_ip_from 2.2.2.xxx; # caravan
real_ip_header proxy_protocol; # using proxy_protocol as header, it cannot be dynamically set
# check CF-Connecting-IP, use $remote_addr as fallback
map $http_cf_connecting_ip $cf_ip_addr {
"" $remote_addr;
default $http_cf_connecting_ip;
}
# dynamic address for further X-Real-IP header passed to backend
map $proxy_protocol_addr $real_ip_addr {
"" $cf_ip_addr;
default $proxy_protocol_addr;
}
# updated log_format, make sure to have it before using in access_log
log_format main '$real_ip_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "
access_log /var/log/nginx/access.log main;
===== nginx.conf OR conf.d/000-realip.conf =====
===== nginx.conf OR conf.d/001-vhost.conf =====
server {
listen 80 ;
listen 81 proxy_protocol;
server_name DOMAIN www.DOMAIN;
location /
{
proxy_pass http://DOMAIN_UPSTREAM;
proxy_set_header Connection "";
proxy_pass_header Set-Cookie;
proxy_set_header Host $host;
proxy_set_header Cookie $http_cookie;
proxy_set_header X-Real-IP $real_ip_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
===== nginx.conf OR conf.d/001-vhost.conf =====
Result (from Nginx logs):
HAProxy->Nginx: 3.3.3.3 - - [22/Feb/2017:23:37:35 +0300] "GET /p.php HTTP/1.1" 200 1340 "-" "curl/7.38.0" "-"
Nginx->Nginx: 2.2.2.2 - - [22/Feb/2017:23:37:49 +0300] "GET /p.php HTTP/1.1" 200 1400 "-" "curl/7.38.0" "3.3. 3.3"
Direct to Nginx: 3.3.3.3 - - [22/Feb/2017:23:37:58 +0300] "GET /p.php HTTP/1.1" 200 1340 "-" "curl/7.38.0" "- "
3.3.3.3 is the real IP of the client, 2.2.2.2 is the IP of the external Nginx.
As you can see, only when using an external Nginx, you cannot override $remote_addr and cannot replace it by using the realip module, because we have it set to proxy_protocol. But even in this case, we still have a backup in the form of $http_x_forwarded_for. This is with regard to the logs (especially since you can also use $real_ip_addr there, which I eventually did). The backend itself now always gets the correct X-Real-IP and its geo-module also works correctly.
I really hope that such a strange case will help someone else;)

Answer 2

Interesting. So far I see only two options. 1) do not use the proxy protocol, but simply put the X-Real-Ip header on haproxy and decrypt it on nginx. 2) Describe two servers on different ports, one for the proxy protocol on a custom port, the other for regular 80/443.