Overriding incoming request IP

[blinkseb]

Hi everyone,

First of all, thanks for this awesome piece of software!

I'm trying to use OpenResty to dynamically update the incoming request IP based on a query parameter. I have an external service which is calling a given url on my server, and pass as argument the real ip. I need to route this request to another route on the same server, while updating the incoming IP. As a simple example, please see the following configuration:

log_format logf '$msec'             # ts
	     '\t$uri'                   # client
	     '\t$remote_addr'           # ip
	     '\t$http_referer'          # page
	     '\t$query_string'          # adcontrol
	     '\t$http_user_agent'       # uagent
	     '\t$server_port'           # port
	     '\t$http_x_forwarded_for'; # proxy

server {
    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;

    listen       80 default_server;
    server_name _;

    error_log hmx_error.log;
    access_log hmx_access.log logf;

    location / {
        access_log off;

        access_by_lua_block {
            ngx.log(ngx.ERR, "Changing IP")
            ngx.req.set_header('X-Forwarded-For', '4.3.2.1')
            ngx.exec("/t")
        }
    }

    location = /t {

    }
}

The service calls the / route, which update the X-Forwarded-For header with the new IP, and the request is forwarded to the /t route. This works exactly as expected, see that the IP is correctly updated:

> curl localhost
nginx acces.log: 1526930357.285	/t	4.3.2.1	-	-	curl/7.54.0	80	4.3.2.1

However, my server is behind a load balancer, and the incoming request already has the X-Forwarded-For header set. In this configuration, it seems like updating the header doesn't do anything:

> curl -H "X-Forwarded-For: 1.2.3.4" localhost
nginx access.log: 1526930549.882	/t	1.2.3.4	-	-	curl/7.54.0	80	4.3.2.1

As you can see, the proxy IP is the correct one, but the remote IP is the one from the original X-Forwarded-For header, and not the one from the updated header. It looks like the real_ip_header handler is not called again when the header is updated.

I think this is an issue since it's working correctly if the header is not set in the first place. If you need more information from me, please just ask!

Again, thanks a lot for this project!

Openresty version (also tested on Linux with the same issue):

nginx version: openresty/1.13.6.2
built by clang 9.1.0 (clang-902.0.39.1)
built with OpenSSL 1.1.0h  27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/Cellar/openresty/1.13.6.2/nginx --with-cc-opt='-O2 -I/usr/local/include -I/usr/local/opt/pcre/include -I/usr/local/opt/openresty-openssl/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/Cellar/openresty/1.13.6.2/luajit/lib -L/usr/local/lib -L/usr/local/opt/pcre/lib -L/usr/local/opt/openresty-openssl/lib' --pid-path=/usr/local/var/run/openresty.pid --lock-path=/usr/local/var/run/openresty.lock --conf-path=/usr/local/etc/openresty/nginx.conf --http-log-path=/usr/local/var/log/nginx/access.log --error-log-path=/usr/local/var/log/nginx/error.log --with-pcre-jit --with-ipv6 --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-dtrace-probes --with-stream --with-stream_ssl_module --with-http_ssl_module

[spacewander]

@blinkseb Could you use rewrite_by_lua_block instead of access_by_lua_block? Nginx sets real ip before the access phase, so change the header in access phase does not work.

[blinkseb]

Hi @spacewander, thanks a lot for your answer. I just tried with rewrite_by_lua_block, but I observe the same behavior. Interestingly, message in error logs already show the X-Forwarded-For ip already assigned as the remote address when entering the rewrite_by_lua_block method.

[spacewander]

@blinkseb Sorry, I missed something in my previous answer.

It seems that Nginx doesn't allow to set the real ip twice: https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_realip_module.c#L152

So change the X-Forwarded-For in the middle could not change the real ip. Maybe you need to find other way to work around it?

[blinkseb]

That's what I was afraid of :disappointed: Indeed, if I comment the check in https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_realip_module.c#L152, the source IP is correctly changed. Any chance it's supported natively by openresty ?

Anyway, thanks a lot for your help and answers!

[spacewander]

@blinkseb

Any chance it's supported natively by openresty

Maybe something else depends on Nginx doesn't allow to set the real ip twice to work correctly. So hack the Nginx source code might cause unexpected result. My suggestion is to write a Nginx C module and call the C API to set connection address via LuaJIT's FFI.

[Darkangeel-hd]

Any update on this issue?

I'm trying to remove the X-Real-IP before sending the request to the upstream server but I cannot find how. I've been searching the web for hours. I'm using Kong, but it uses OpenResty, so a valid solution here is also valid on the other.

Any help is welcome and appreciated

[spacewander]

@blinkseb

Any chance it's supported natively by openresty

Maybe something else depends on Nginx doesn't allow to set the real ip twice to work correctly. So hack the Nginx source code might cause unexpected result. My suggestion is to write a Nginx C module and call the C API to set connection address via LuaJIT's FFI.

I developed a C API to do this, see: https://github.com/apache/apisix/blob/2f7833e07479610ed5c8148399c347df85c20b96/apisix/plugins/real-ip.lua#L169 https://github.com/api7/apisix-nginx-module/blob/3886413f88e93f58afe399aa27af6e04fb32c35c/lib/resty/apisix/client.lua#L55