[fleetd] Agent stuck in a crash loop

[ksatter]

fleetd version:v1.24.0 Host OS: Windows server 2016

---

💥  Actual behavior

Hosts with fleetd installed are either not enrolling in Fleet or are showing as online, but unable to be fetched.

In the fleetd logs, the following errors are present:

Host A:

2024-05-22T13:25:22-04:00 INF Service Interrogate Requested
2024-05-22T13:25:42-04:00 ERR getHostInfo via osquery output="[\r\n  {\"hardware_serial\":\"asdfasdfc5\",\"hostname\":\"[email protected]\",\"instance_id\":\"asdfasdfasdf\",\"platform\":\"windows\",\"uuid\":\"asdfasdfasdf\"}\r\n]\r\n" stderr=
2024-05-22T13:25:42-04:00 ERR run orbit failed error="get UUID: exit status 78"

2024-05-22T13:25:18-04:00 INF early update check failed error="update metadata: update metadata: tuf: failed to download 5.root.json: Get \"https://tuf.fleetctl.com/5.root.json\": read tcp IP-> IP: wsarecv: An existing connection was forcibly closed by the remote host."
2024-05-22T13:25:18-04:00 ERR update metadata before getting components error="update metadata: tuf: failed to download 5.root.json: Get \"https://tuf.fleetctl.com/5.root.json\": read tcp 10.62.190.13:54368->169.150.236.99:443: wsarecv: An existing connection was forcibly closed by the remote host."

Host B:

W0515 14:58:44.916077  6736 windowseventlogpublisher.cpp:129] Failed to subscribe to true: 15007
W0515 14:58:45.690688  3960 bitlocker_info.cpp:52] Error retreiving information from WMI.
I0515 14:58:48.947904  3960 interfaces.cpp:102] Failed to retrieve network statistics for interface 7
I0515 14:58:49.026556  3960 interfaces.cpp:102] Failed to retrieve network statistics for interface 1
I0515 14:58:49.065276  3960 interfaces.cpp:130] Failed to retrieve physical state for interface 1
I0515 14:58:49.076035  3960 interfaces.cpp:157] Failed to retrieve DHCP and DNS information for interface 1
I0515 14:58:49.092772  3960 interfaces.cpp:102] Failed to retrieve network statistics for interface 6
2024-05-15T14:58:51-04:00 INF Service Interrogate Requested
2024-05-15T14:59:14-04:00 INF Service Interrogate Requested
2024-05-15T14:59:52-04:00 INF Service Interrogate Requested
2024-05-15T15:00:15-04:00 INF Service Interrogate Requested
2024-05-15T15:00:52-04:00 INF Service Interrogate Requested
2024-05-15T15:01:25-04:00 INF Service Interrogate Requested
2024-05-15T15:01:46-04:00 INF Service Interrogate Requested
2024-05-15T15:01:58-04:00 INF Service Interrogate Requested
2024-05-15T15:02:25-04:00 INF Service Interrogate Requested
2024-05-15T15:02:59-04:00 INF Service Interrogate Requested
2024-05-15T15:03:26-04:00 INF Service Interrogate Requested
[repeating]

In Windows event logs, errors indicate that the service is being restarted due to a crash.

More log information is available in the Slack thread

🧑‍💻  Steps to reproduce

1. TODO
2. TODO

🕯️ More info (optional)

N/A

[zayhanlon]

Additional information from the customer that may be helpful:

2 days ago, the customer began a test enrollment with 2 Windows workstations and 3 windows test servers enrolled. All worked well

They decided to deploy 19 production DC’s and none of them were speaking back to Fleet

They asssumed a firewall issue. Got the port opened, but still no dice. They then spent two days so far trying to figure out why Fleet wasn't running / speaking to these devices. 

They commented that they think it could be two bugs. One related to the server version and another one on the agent side

[zayhanlon]

Closed in error

[ksatter]

Thanks @zayhanlon

[zayhanlon]

@sharon-fdm will this make it into 4.51?

[sharon-fdm]

@zayhanlon this will be part of the next Fleetd (agent) release 1.26.0 that we plan to be close in time to 4.51.0 (early next week) cc: @lucasmrod

[fleet-release]

Crash loop fixed, peace, Fleet now smoothly sails the cloud, Secure, swift, unceased.