help request: Route traffic through company proxy / firewall

[nareshnagamalle]

Description

Dear Sir/Madam,

Sub: Route traffic through company proxy / firewall

I have deployed apisix 2.15.0. And followed the steps mentioned in https://apisix.apache.org/docs/apisix/2.14/getting-started/. The following i have done step1: curl http://localhost:9080/apisix/admin/routes -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X POST -d ' { "name": "Route to httpbin", "uris": ["/*"], "upstream": { "type": "roundrobin", "nodes": { "httpbin.org": 1 } } }' After above step, the successfull expected output has come step 2: curl 'localhost:9080/anything?foo=bar&baz' -X POST -d '{ "hello": "world" }' -H 'Content-Type: application/json'

After Step2, expected result has not come, it is showing the following message

502 Bad Gateway

---

openresty

I have already enabled our company proxy details in .bashrc and profile.

Please help to resolve the issue.

I have already searched earlier issues and found the same issue in the following url https://github.com/apache/apisix/issues/2280, but there is no answer for this.

Kindly help tor resolve. Please let me know any other information required from my side

Thanks & Regards, Naresh.N

Environment

2.15.0- APISIX version (run apisix version): Cent OS- Operating system (run uname -a): 1.21.4.1- OpenResty / Nginx version (run openresty -V or nginx -V): 3.5.4- etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):

• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tzssangglass]

It looks more like your company's network environment (egress traffic whitelist) is causing the problem?

[soulbird]

Try:

curl 'localhost:9080/anything?foo=bar&baz' -X POST -d '{ "hello": "world" }' -H 'Content-Type: application/json' -H"host: httpbin.org"

[nareshnagamalle]

We tried above things, it is not working.

Even I am unable to connect localhost upstream and not getting what to do?

When I am seeing access log in "/usr/local/apisix/logs"

127.0.0.1 - - [01/Sep/2022:16:44:47 +0530] 127.0.0.1 "GET /index.html HTTP/1.1" 504 164 60.270 "-" "curl/7.29.0" 3.94.154.124:80 504 60.000 "http://127.0.0.1"

I don't know from where this IP is coming????

I have used below command to create route for local upstream

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d ' { "methods": ["GET"], "host": "127.0.0.1", "uri": "/*", "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:80": 1 } } }'

All helps are appreciated.

[soulbird]

You initially created the upstream using httpbin.org, this IP is the result of DNS resolution. You're better off using a local service to create the upstream to avoid network problems that plague your tests.

[nareshnagamalle]

That's what we have tried and same I posted before this post.

For localhost upstream also apisix route is not working.

5XX error is coming.

[tokers]

Obviously, you should check out if you can access httpbin.org directly without the proxy to make sure the machine you deployed APISIX can connect to httpbin.org.

[tzssangglass]

You need to verify this

1. on the machine where APISIX is deployed, access upstream via curl (httpbin.org or 127.0.0.1:80) and verify that it is accessible and the network is OK
2. configure the APISIX upstream to be the upstream you just accessed via curl, access APISIX via curl, and verify that APISIX can proxy the request to the upstream.

[nareshnagamalle]

Thanks @tokers and @tzssangglass for your responses.

The machine which I deployed apisix , following are tested.

1. When I accessed httpbin.org or any other external portal with curl command, the response is coming fine. Reason I have given my company proxy details in .bashrc and .bash_profile, so it could able to fetch our proxy and send request to external portal and getting response.
2. I have done the following i. curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d ' { "methods": ["GET"], "host": "example.com", "uri": "/anything/", "upstream": { "type": "roundrobin", "nodes": { "httpbin.org:80": 1 } } }' Response: Got successful response {"node":{"value":{"create_time":1662109538,"priority":0,"methods": ["GET"],"update_time":1662351433,"uri":"/anything/","host":"example.com","status":1,"upstream":{"nodes":{"httpbin.org:80":1},"hash_on":"vars","scheme":"http","pass_host":"pass","type":"roundrobin"},"id":"1"},"key":"/apisix/routes/1"},"action":"set"}

ii. After above step, executed the following statement curl -i -X GET "http://127.0.0.1:9080/anything/foo?arg=10" -H "Host: example.com" Response Got following error response HTTP/1.1 504 Gateway Time-out Date: Mon, 05 Sep 2022 04:21:24 GMT Content-Type: text/html; charset=utf-8 Content-Length: 164 Connection: keep-alive Apisix-Plugins: no plugin Server: APISIX/2.15.0 X-APISIX-Upstream-Status: 504

504 Gateway Time-out

---

openresty **I feel reason for above error response** is apisix software not considering our company proxy details from .bashrc and bash_profile file and not able to connect to external portal .

And please find also the messages from access.log and error.log access.log 127.0.0.1 - - [05/Sep/2022:09:51:24 +0530] example.com "GET /anything/foo?arg=10 HTTP/1.1" 504 164 60.038 "-" "curl/7.29.0" 34.227.213.82:80 504 60.000 "http.example.com" error.log 2022/09/05 09:51:24 [error] 3702#3702: *9444121 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: _, request: "GET /anything/foo?arg=10 HTTP/1.1", upstream: "http://34.227.213.82:80/anything/foo?arg=10", host: "example.com"

Please let me know where can I give our company proxy details in apisix software, so that when ever apisix want to communicate with external portal it should consider our company proxy and connect &get response.

Please let me know any other details are required further and help to resolve the issue

[tzssangglass]

Please let me know where can I give our company proxy details in apisix software, so that when ever apisix want to communicate with external portal it should consider our company proxy and connect &get response.

Please let me know any other details are required further and help to resolve the issue

There is no such function now

[kingluo]

@nareshnagamalle apisix doesn't support sending the request to upstream via external proxy server yet.

What's your type of proxy server? socks5? If so, you could make a trick with iptables and tcpsocks to achieve your goal.

Example:

# Here I use ssh to create a socks5 server as demo
# in your case, you should use your real proxy instead
ssh -o ServerAliveInterval=60 -N -D 127.0.0.1:30000 <username>@<ssh server> -p 20022 &

# compile and run tcpsocks
cd /opt
git clone https://github.com/vi/tcpsocks
cd tcpsocks
make
./tcpsocks 0.0.0.0 12345 REDIRECT REDIRECT 127.0.0.1 30000

# in another terminal
# setup iptables rules
iptables -t nat -A QQQ -p tcp -d 54.147.68.244 -j REDIRECT --to-ports 12345
iptables -t nat -I OUTPUT 1 -j QQQ
iptables -t nat -I PREROUTING 1 -j QQQ


# setup a rule in apisix
curl http://127.0.0.1:9080/apisix/admin/routes/route_via_proxy  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "/anything",
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "54.147.68.244": 1
        }
    }
}'

# check if it works
curl -i http://127.0.0.1:9080/anything
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 386
Connection: keep-alive
Date: Wed, 07 Sep 2022 04:06:54 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Server: APISIX/2.15.0
...

# check tcpsocks logs
xxx:55646 -> 54.147.68.244:80 [5->6]
    54.147.68.244:80 -> xxx:55646 [6->5] Started
    54.147.68.244:80 -> xxx:55646 [6->5] 616:214 Finished

Noe that 54.147.68.244 is one of the resolved ip address of httpbin.org. So the shortage of this way is you could only use ip address to access your upstream.

[tzssangglass]

This only solves the 54.147.68.244 problem, but there may actually be many nodes (many upstream ip).

[kingluo]

@tzssangglass With ipset, it's an easy job.

Example:

# use domain to define the route
curl http://127.0.0.1:9080/apisix/admin/routes/route_via_proxy  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "/anything",
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "httpbin.org": 1
        }
    }
}'

# create an ipset
ipset create myset hash:net

# setup iptables rules
iptables -t nat -A QQQ -p tcp -m set --match-set myset dst -j REDIRECT --to-ports 12345
iptables -t nat -I OUTPUT 1 -j QQQ
iptables -t nat -I PREROUTING 1 -j QQQ

Run this script to update ipset periodically:

https://gist.github.com/kingluo/8944c1435c3c93fd7bccaca3f4f810ba

Adjust hosts to contain your upstream domains.

[tzssangglass]

OK, there is a little off-topic.

Always I think this is not a problem with APISIX and APISIX does not do such things.

[jujiale]

apisix doesn't support sending the request to upstream via external proxy server yet.

hello，I want to know why apisix doesn't support sending the request to upstream via external proxy server yet.

[jujiale]

Please let me know where can I give our company proxy details in apisix software, so that when ever apisix want to communicate with external portal it should consider our company proxy and connect &get response. Please let me know any other details are required further and help to resolve the issue

There is no such function now

@tzssangglass hello, I want to know why such function is not support, and as @kingluo say,why apisix doesn't support sending the request to upstream via external proxy server yet.