orbstack
orbstack copied to clipboard
orbstack
IP & domain access not working with Surge TUN mode
Describe the bug
ip & domain direct access not working with surge's tun mode VPN (https://nssurge.com)
turn off the TUN mode works fine
To Reproduce
- start a nginx container
- access via ip or domain
- not working
Expected behavior
No response
Diagnostic report (required)
OrbStack info: Version: 0.16.0 (160000) Commit: b92cc5239da462d9918d844fc5b04ec110a35e1a (v0.16.0)
System info: macOS: 13.4.1 (22F82) CPU: arm64, 12 cores CPU model: Apple M2 Max
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-08-11T07-29-11.454063Z.zip
Screenshots and additional context (optional)
No response
Can you try:
traceroute container.orb.local
curl -v container.orb.local
curl -v -4 container.orb.local
curl -v -6 container.orb.local
... and share the output of each command?
I have one like this - reproducible.
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit
INFO[2023/08/11 09:11:16] [smtpd] starting on 0.0.0.0:1025
INFO[2023/08/11 09:11:16] [http] starting server on http://localhost:8025/
http://localhost:8025/ is available and works well.
When selecting "Open in browser" in Orbstack it opens in Safari to http://brave_goldwasser.orb.local and just hangs there
~/Sites traceroute http://brave_goldwasser.orb.local
^C
~/Sites traceroute brave_goldwasser.orb.local
traceroute to brave_goldwasser.orb.local (192.168.215.2), 64 hops max, 52 byte packets
1 192.168.215.2 (192.168.215.2) 3.157 ms 1.090 ms 0.583 ms
~/Sites curl -v http://brave_goldwasser.orb.local
* Trying 192.168.215.2:80...
* Connected to brave_goldwasser.orb.local (192.168.215.2) port 80 (#0)
> GET / HTTP/1.1
> Host: brave_goldwasser.orb.local
> User-Agent: curl/8.1.2
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
~/Sites curl -v -4 http://brave_goldwasser.orb.local
* Trying 192.168.215.2:80...
* Connected to brave_goldwasser.orb.local (192.168.215.2) port 80 (#0)
> GET / HTTP/1.1
> Host: brave_goldwasser.orb.local
> User-Agent: curl/8.1.2
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
~/Sites curl -v -6 http://brave_goldwasser.orb.local
* Trying [fd07:b51a:cc66:1:0:242:c0a8:d702]:80...
* Connected to brave_goldwasser.orb.local (fd07:b51a:cc66:1:0:242:c0a8:d702) port 80 (#0)
> GET / HTTP/1.1
> Host: brave_goldwasser.orb.local
> User-Agent: curl/8.1.2
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
~/Sites
@PhilETaylor That's a different issue. Mailpit's SMTP server on port 1025 trips up the auto port detection. It's been improved for the next version, but in the meantime you can use http://brave_goldwasser.orb.local:8025.
yeah I see that now that I stopped the container after sending the last message, I returned to the browser and I see
220 519b0ca15e80 Mailpit ESMTP Service ready
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
500 5.5.2 Syntax error, command unrecognized
which confirms what you said, it was bound to the wrong open port :)
If unrelated to the OP then sorry for the noise.
Hi, here is the log. It appears that there is a problem when enabling TUN VPN. Should I exclude the *.orb.local DNS query from the VPN?
nssurge offers a 14-day trial version, and several VPNs use TUN mode to capture all data packets from the entire system.
➜ ~ traceroute container.orb.local
traceroute to container.orb.local (198.18.3.187), 64 hops max, 52 byte packets
1 198.18.0.1 (198.18.0.1) 0.761 ms 0.334 ms 0.152 ms
2 * * *
3 * * *
4 * * *
^C
➜ ~ curl -v container.orb.local
* Trying 198.18.3.187:80...
* Connected to container.orb.local (198.18.3.187) port 80 (#0)
> GET / HTTP/1.1
> Host: container.orb.local
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
➜ ~ curl -v -4 container.orb.local
* Trying 198.18.3.187:80...
* Connected to container.orb.local (198.18.3.187) port 80 (#0)
> GET / HTTP/1.1
> Host: container.orb.local
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
➜ ~ curl -v -6 container.orb.local
* Closing connection 0
curl: (7) Couldn't connect to server
➜ ~
➜ ~ scutil --dns
DNS configuration
resolver #1
search domain[0] : localdomain
nameserver[0] : 198.18.0.2
flags : Request A records
reach : 0x00000002 (Reachable)
resolver #2
domain : local
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300000
resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300200
resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300400
resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300600
resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300800
resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 301000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : localdomain
nameserver[0] : 198.18.0.2
if_index : 14 (en0)
flags : Scoped, Request A records
reach : 0x00000002 (Reachable)
after bypass the 192.168.0.0/16 in VIF Excluded Routers, ip direct access works, but domain access still fail.
➜ ~ traceroute clever_heisenberg.orb.local
traceroute to clever_heisenberg.orb.local (198.18.3.156), 64 hops max, 52 byte packets
1 198.18.0.1 (198.18.0.1) 0.828 ms 0.253 ms 0.172 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * *^C
➜ ~ traceroute 192.168.215.2
traceroute to 192.168.215.2 (192.168.215.2), 64 hops max, 52 byte packets
1 192.168.215.2 (192.168.215.2) 1.373 ms 0.923 ms 0.371 ms
➜ ~
➜ ~
➜ ~
➜ ~ curl -v clever_heisenberg.orb.local
* Trying 198.18.3.156:80...
* Connected to clever_heisenberg.orb.local (198.18.3.156) port 80 (#0)
> GET / HTTP/1.1
> Host: clever_heisenberg.orb.local
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
➜ ~ curl -v 192.168.215.2
* Trying 192.168.215.2:80...
* Connected to 192.168.215.2 (192.168.215.2) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.215.2
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Fri, 11 Aug 2023 13:56:29 GMT
Yes, I have already added *.orb.local to the bypass proxy settings.
After further testing, it appears that Safari is functioning properly. However, Chrome and Edge (OSX) are not working.
@te-simonren Can you run all the commands again and share the output, along with a new diagnostic report?
here are the logs and diagnostic report:
OrbStack info: Version: 0.16.1 (160100) Commit: 4235b341267c814080ec0e117d1e680d8f8f8676 (v0.16.1)
System info: macOS: 13.5.1 (22G90) CPU: arm64, 12 cores CPU model: Apple M2 Max
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-08-18T06-45-18.172002Z.zip
➜ ~ traceroute clever_heisenberg.orb.local
traceroute to clever_heisenberg.orb.local (192.168.215.2), 64 hops max, 52 byte packets
1 192.168.166.1 (192.168.166.1) 5.554 ms 2.373 ms 2.511 ms
2 192.168.0.1 (192.168.0.1) 2.847 ms 3.089 ms 3.570 ms
3 * *^C
➜ ~ curl -v clever_heisenberg.orb.local
* Trying [fd07:b51a:cc66:0:a617:db5e:c0a8:d702]:80...
* Connected to clever_heisenberg.orb.local (fd07:b51a:cc66:0:a617:db5e:c0a8:d702) port 80 (#0)
> GET / HTTP/1.1
> Host: clever_heisenberg.orb.local
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Fri, 18 Aug 2023 06:40:54 GMT
< Content-Type: text/html
< Content-Length: 8702
< Last-Modified: Thu, 22 Dec 2022 20:49:18 GMT
< Connection: keep-alive
< ETag: "63a4c2ce-21fe"
< Accept-Ranges: bytes
➜ ~ curl -v -4 clever_heisenberg.orb.local
* Trying 192.168.215.2:80...
^C
➜ ~ curl -v -6 clever_heisenberg.orb.local
* Trying [fd07:b51a:cc66:0:a617:db5e:c0a8:d702]:80...
* Connected to clever_heisenberg.orb.local (fd07:b51a:cc66:0:a617:db5e:c0a8:d702) port 80 (#0)
> GET / HTTP/1.1
> Host: clever_heisenberg.orb.local
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Fri, 18 Aug 2023 06:41:39 GMT
< Content-Type: text/html
< Content-Length: 8702
< Last-Modified: Thu, 22 Dec 2022 20:49:18 GMT
< Connection: keep-alive
< ETag: "63a4c2ce-21fe"
< Accept-Ranges: bytes
<
➜ ~ scutil --dns
DNS configuration
resolver #1
search domain[0] : localdomain
nameserver[0] : 114.114.114.114
flags : Request A records
reach : 0x00000002 (Reachable)
resolver #2
domain : local
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300000
resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300200
resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300400
resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300600
resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300800
resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 301000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : localdomain
nameserver[0] : 114.114.114.114
if_index : 15 (en0)
flags : Scoped, Request A records
reach : 0x00000002 (Reachable)
Can't reproduce. Chrome seems to cache DNS answers for longer than macOS does, so it's likely that you had the bad (intercepted) addresses cached. Maybe try again now?
I have tried troubleshooting Chrome, but it is still not working. However, Safari and Firefox are functioning properly. I believe there may be a misconfiguration issue with my Chrome. Thank you for your response and support.
Guys, it's working now. I was able to fix the issue by disabling and then re-enabling network access to the container domain and IPs. After each action, I made sure to reopen the orbstack.
hi @kdrag0n, safari works because it's using ipv6 address. with recent v1.0 of Orbstack, the ip/domain access failed when enable VPN-TUN mode. turn off VPN-TUN mode everything work fine.
@te-simonren For VPN-TUN mode, do you mean enable the "Enhanced Mode"?
I tried the following situations:
With "Enhanced Mode" enabled
- In Safari, visiting domain works without any additional configuration in Surge
- In Chromium based browser, visiting domain cannot work
With "Enhanced Mode" disabled
- Both browsers work properly.
Yes, exactly as you said. When Enhanced Mode is enabled, I believe Safari functions properly because it utilizes IPv6. To confirm that IPv4 is not functioning, you can use the commands curl -4 and curl -6.
@te-simonren The problem seems caused by Surge's VIF taking over the OrbStack's virtual network. So when Enhanced Mode is enabled, the DNS queries are sent to Surge DNS address (198.18.0.2), but it returns a fake IP address (e.g. 198.18.1.181) instead of the container's IP address.
@goofansu It's possible, but even when using the IP copied from the orbstack UI, it still doesn't work with Enhanced Mode enabled.
Here is a log of the routing table with/without surge's enhanced mode. There is no bridge101 when enhanced mode enabled.
@te-simonren I found the solution: set always-real-ip to *.orb.local, so that Surge sends the DNS query to a real DNS server for *.orb.local domains in Enhanced Mode.
Notice: remove all orb.local related configuration beforehand. I only set this config for domain works.
In Surge GUI, this config locates at "More -> Settings -> Advanced".
After that, the dig <container>.orb.local command can find the correct DNS record, and visiting domain works correctly.
❱ dig docker-getting-started.orb.local
; <<>> DiG 9.10.6 <<>> docker-getting-started.orb.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 49457
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;docker-getting-started.orb.local. IN A
;; AUTHORITY SECTION:
. 3542 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2023092400 1800 900 604800 86400
;; Query time: 21 msec
;; SERVER: 198.18.0.2#53(198.18.0.2)
;; WHEN: Sun Sep 24 20:57:19 CST 2023
;; MSG SIZE rcvd: 136
@goofansu, yes, you are correct!
Furthermore, I take additional steps as follows:
- Disable the "Enhanced Mode" setting.
- Launch a pod and Kubernetes with ingress from Orbstack.
- Verify the output of
netstat -nr | grep 192to ensure that you have obtained routes for both192.168.194and192.168.215. - Enable the "Enhanced Mode" setting.
- Ensure that the pod and Kubernetes (k8s) are kept running. If they stop, routing records will be deleted.
By following these steps, everything should function properly.
@te-simonren Thank you for the details, I didn't use Kibernetes😄 Can you visit the IP address in browser directly? I cannot, even when Surge is closed, but visiting domain is enough for me.
@goofansu, both the IP and domain are working for me. Try rebooting Orbstack after closing Surge.
@te-simonren Thank you, both domain and IP work now with Surge TUN Mode.
Ref: https://github.com/orbstack/orbstack/issues/529#issuecomment-1732565991 https://github.com/orbstack/orbstack/issues/529#issuecomment-1732875282
@kdrag0n It's not a bug in OrbStack, issue can be closed. Thanks
@kdrag0n,
Yes, after turning off Surge and restarting the PostgreSQL pod, everything started running smoothly.
This is my test result,
-
Start orbstack first, then start Surge.
-
Start Surge first, then start orbstack.
OrbStack info: Version: 1.1.0 Commit: bfae870a3b9d4fba5c1c26711ac5a5b9183bdb74 (v1.1.0)
System info: macOS: 13.6.3 (22G430) CPU: amd64, 8 cores CPU model: Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz Model: MacBookPro14,3 Memory: 16 GiB
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-11-22T01-07-06.125951Z.zip
In the second case, the local service can not be accessed through domain & IP
Seems that orbstack won't add a route rule for the connection if surge enhance mode is enabled
This is my test result:
No Surge enabled, a new route rule is added
$ docker run --rm -d --name nginx nginx
$ curl -v https://nginx.orb.local
* Trying 192.168.215.2:443...
* Connected to nginx.orb.local (192.168.215.2) port 443
$ netstat -rn
__REDACTED__
192.168.215 link#26 UC bridge101 !
__REDACTED__
If surge is enabled, no new route rule will be added, so the default route is used
13:06:23.158618 [Connection] Handled by VIF
13:06:23.158923 [TLS] TLS Client Hello SNI: nginx1.orb.local
13:06:23.163335 [Rule] Sub-rule matched: IP-CIDR 192.168.0.0/16(LAN)
13:06:23.163367 [Rule] Rule matched: RULE-SET LAN
13:06:23.164336 [Socket] Connecting with address: 192.168.215.2, bound to the primary interface (en0) explicitly under Enhanced Mode
@kdrag0n I noticed that you closed my #1002 . You think my problem is the same as #529, but I just checked my Surge configuration and found that I have enabled the enhanced mode of Surge and can also access *.orb.local. It seems that my problem may not be caused by this reason.
I think TUN mode is what I call enhanced mode (Chinese translation is enhanced mode)
@te-simonren Friend, you only need to configure Surge simply to access *.orb.local normally when tun mode is enabled. Configuration: Always Real IP Hosts: *orb.local
早知道和你说中文好了
Screenshot as follows
