fleet
fleet copied to clipboard
fleetdm
VPN IP address isn't showing up
Fleet version: Fleet 4.44.0
💥 Actual behavior
If a host connects to the internet via a VPN, Fleet doesn't show the IP address for this host.
🧑💻 Steps to reproduce
- Enroll a host to Fleet that connects to the internet via a VPN
- Observe the empty "Private IP" and "Public IP" field on the host's Host details page.
🕯️ More info (optional)
Today, the Fleet filters out private IP addresses in the public IP field. We added this filtering to address the following issue: https://github.com/fleetdm/fleet/issues/8924
The VPN address is one of these private IP's that gets filtered out.
When Fleet filters out a public IP this is what the Host details page looks like this:
The "Learn more" links to this page in the docs: https://fleetdm.com/docs/deploy/public-ip
🛠️ To fix
- [ ] Backend: Remove the filtering for the public IP field so that the IP address the host uses to connect to Fleet is always shown in the Public IP field in the UI/API.
- In the VPN IP address case, this will reveal the VPN IP address.
- [ ] Frontend: For the Public IP address field on the Host details page and Hosts page, always show a tooltip w/ the dotted line under the "Public IP address" header:
The IP address the host uses to connect to Fleet.- On the Host details and Host page, remove the old tooltip on hover over the value:
- [ ] Remove the purple callout box from this doc page: https://fleetdm.com/docs/deploy/public-ip
Hi @kswagler-rh!
Just for reference, #8924 is the issue where we introduced the behavior to not allow "Private IPs" in the "Public IP address".
I'm assigning this to @zayhanlon for triage in the CX group. This may require some thought, as we explicitly disallowed private IPs in the public IP field in #8924 as a bug fix.
Another customer feedback:
I think having an IP address with no extra distinction placed upon it and having Fleet do its best to record that. I don't care about public vs. public. I.e. go back to thinking of it in terms of primary_ip
At our organization, it is quite common for there to be two proxies between nodes and Fleet. This does make it quite hard to rely on HTTP headers for valid info. This is a popular and decent summary of those issues: https://www.brainonfire.net/blog/2022/03/04/understanding-using-xff/
@sharon-fdm
@xpkoala I think @sharon-fdm meant to nudge you for specific reproduction steps instead of @xp.
@xpkoala tagged frontend for this bug, however, the value being returned from the API for IP addresses is a backend issue and there is no logic modifying these values on the frontend. Therefore, I am removing the ~frontend tag and adding ~backend.
From reading through the comments, it looks like the CX managers should have a conversation where/how a VPN IP address will present itself in the host data and then assign this ticket to the backend. It'll also be super helpful for QA to outline reproduction steps for dev/testing purposes.
Hey @dherder this looks like a bug in Fleet. When you get the chance, can you please update the issue description to use the bug report template: https://github.com/fleetdm/fleet/issues/new?assignees=&labels=bug%2C%3Areproduce%2C%3Aincoming&projects=&template=bug-report.md&title=
After we update the description, can you please assign @sharon-fdm, adding the :release, incoming, and #g-endpoint ops so that it ends up on the endpoint ops board.
@kswagler-rh thanks for calling this to our attention again! Sorry you're still running into this.
@kswagler-rh im testing in Fleet 4.43.0 and cannot reproduce. My VPN IP is being reported in Fleet as the Public IP.
@kswagler-rh I'm assigning this to you to make sure it doesn't slip through the cracks.
@dherder is unable to reproduce on the latest. Would you please see if you can still reproduce on 4.44.0? If so, please follow Noah's steps (or @dherder) to get this into the Endpoint Ops board so we can fix. This is now our oldest bug in the system, so we want to triage quickly.
Thanks!
Thanks folks. We are a few versions behind, I'll try and get them updated today. Thanks!
Hi all,
We are now running on 4.44.0 for a few days and I do not see changes, we are still not seeing the Public IP field populate with private IPs. I also noticed that the Public IP docs indicate that the field would not show private IP space IPs.
@kswagler-rh here's the steps that I'm taking to try and repo:
Tested on a macOS system running 14.2.1 (23C71)
- Signup for a free VPN service like protonVPN (which is just wireguard I think)
- Confirm that my Public IP reported in Fleet is my WAN (ISP) address
- Get an active VPN connection
- Refetch host vitals
- Confirm Public IP as viewed in Fleet changes from the ISP provided WAN address to the VPN routed address.
Not sure if this could be dependent on the VPN provider? Were you testing on a macOS, Linux, or Windows host?
ah I see the confusion. We are talking about the corporate VPN which hands out 10.0.0.0/8 IPs on the same "segment" as the Fleet server. If you connect to a VPN or on the same network as the Fleet server you would see something like this still
@zayhanlon This does appear to still be the case.
@kswagler-rh Would you please re-post the image from inside GitHub? If it's posted inside ZenHub it doesn't copy over. Thanks!
@xpkoala Would you please try reproducing using the information @kswagler-rh provides? Thanks!
Hey @xpkoala if the bug is unclear, please feel free to schedule some time w/ myself and @kswagler-rh.
We can probably get to the bottom of this in 20 mins together on a call.
@sharon-fdm Will you please make sure this bug is actioned? It's currently our oldest open bug, so we'd like to prioritize getting it resolved. I'm adding this to the release board so it gets more eyes.
@lukeheath , it's on our board now. will try to swap some backend task with it this sprint.
We added this filtering a while ago to fix https://github.com/fleetdm/fleet/issues/9857: https://github.com/fleetdm/fleet/blob/2c383a060fab66105361f7ed88bc9969a02ec5e1/server/service/osquery_utils/queries.go#L376-L389
We may need to define what the "Public IP" field in Fleet is. Is it the IP of the device when it connects to Fleet? Or is it the IP of the device when surfing the internet?
UPDATE: We landed on a solution and updated the "To fix" section in the issue. For more info on the call see this Google doc here (internal).
@sharon-fdm FYI I pulled this bug onto the drafting board and off release board.
Lucas, Rachel, and I are meeting this afternoon to discuss the solution.
Hey @kswagler-rh heads up, I updated the issue description and moved your original issue description here:
Goal
To have the ability to select if private IPs are shown in the public_ip field in the host details.
Context
It will be helpful to be able to quickly and in one place determine the IP that is reaching the Fleet server. An example is when devices are connected to a VPN, and so have a NAT'd Private IP, but are reaching the Fleet server from a private VPN IP.
=========== OP above =========
Mo: I chatted with OP and learned that the the VPN-provided 10.0.0.0/8 IP address is not showing up at all.
Expected Behavior
As a Fleet user who is connecting to the internet via VPN, I would expect my device to show the VPN IP address (special address) in the host details as the public address
Actual Behavior
It does not appear.
From the "To fix" section:
Frontend: For the Public IP field on the Host details page and Hosts page, always show a tooltip to that explains that this is the IP the host uses to connect to Fleet. Points the user to the docs page: https://fleetdm.com/docs/deploy/public-ip
- TODO Rachael: Wireframes for the new tooltip.
@rachaelshaw I passed this bug that needs some UI design to you.
When you get the chance, can you please take a pass at the tooltip placement/UI? During design review, we can riff on the language.
Hey @lucasmrod, @RachelElysia, and @rachaelshaw I updated the "To fix" section in the issue description w/ the expected frontend and backend behavior.
I also think we can remove the Public IPs doc page entirely to cut down on doc content. This is included in the "To fix" section.
Moving this over to the release board.
Backend: Remove the filtering for the public IP field so that the IP address the host uses to connect to Fleet is always shown in the Public IP field in the UI/API. In the VPN IP address case, this will reveal the VPN IP address.
Frontend: For the Public IP address field on the Host details page and Hosts page, always show a tooltip w/ the dotted line under the "Public IP address" header: The IP address the host uses to connect to Fleet.
Hey @terjekv heads up, we're updating the behavior of the Public IP address field.
What do you think?
When you filed this issue, it seemed like we weren't being clear as to how Fleet determines the public IP. Goal of this fix is to make this clear: The IP address the host uses to connect to Fleet.
I also think we can remove the Public IPs doc page entirely to cut down on doc content. This is included in the "To fix" section.
I believe it's still useful for administrators to know the HTTP headers Fleet uses to determine the hosts' IP, right? That said, as part of this change we should remove the WARNING message in that page.
I believe it's still useful for administrators to know the HTTP headers Fleet uses to determine the hosts' IP, right? That said, as part of this change we should remove the WARNING message in that page.
This is correct. While AWS ALB will have one of these headers set, if the administrator is using something else to do handle ingress/load balancing such as nginx, the proxy pass or equivalent will need to set one of this headers in the configuration.
While I could guess that these are what is used (they are the common headers), I would feel better seeing it in the documentation.
