Fleetctl failing to return any results or any debug info

[vic-son]

I'm trying to run fleetctl to extract queries on the command line but a large proportion of the time, the queries are not displaying any results. Sometimes they will work but certainly not consistently or reliably.

Our fleet server is installed in multiple docker containers managed via a load balancer.

I've seen other issues raised where fleetctl has displayed debug information but even when I use the --debug option I do not get any details.

What version of fleet are you using (fleet version --full)?

fleet - version 2.0.2
  branch:       changelog
  revision:     96008f11bb51605cbf22b7bb6f883d262ce13a28
  build date:   2019-01-18T00:27:22Z
  build user:   zwass
  go version:   go1.11.3

What operating system are you using?

Oracle Linix 6.8

What did you do?

$ fleetctl query --timeout 25m --query 'select name,version from os_version' --labels 'All Hosts' --exit

What did you expect to see?

Query results

What did you see instead?

[----------]$ fleetctl query --timeout 25m --query 'select name,version from os_version' --labels 'All Hosts' --exit ⠋ [----------]$ [----------]$ fleetctl query --timeout 25m --query 'select name,version from os_version' --labels 'All Hosts' 0% responded (0% online) | 0/0 targeted hosts (0/0 online) [----------]$

[kp625544]

ping! same issue with me.

[zwass]

We have been discussing this issue on the osquery Slack (https://osquery.slack.com/archives/C1XCLA5DZ/p1551977973119600) and not yet come to any conclusion.

For anyone else experiencing it, can you please answer the following:

1. Fleet version and fleetctl version
2. Do the hosts appear online in the web UI?
3. Are you able to query via the web UI?
4. Are you able to query a single host through fleetctl?
5. Can you create a new label with the query select 1 and then query against that label via fleetctl?

[zwass]

If anyone is able to provide more information I am interested in debugging this.

[n4ll3ec]

same issue +1. And more detail information for you to debugging

1. fleet & fleetctl version

# fleet version
fleet version 2.0.2

# fleetctl - version 2.0.2
  branch:       master
  revision:     8ca0358bf28173685815b79d8683a4239d629a14
  build date:   2019-01-18T00:39:59Z
  build user:   zwass
  go version:   go1.11.3

2. host is online in the Web UI

3. I can query correctly via the WebUI

4. can't query correctly a single host via fleetctl

5. still failed

[kp625544]

+1 still the same issue

[zwass]

@fl4nker is your Fleet instance possibly behind some load-balancer or other network configuration that does not support websockets? Can you please take a screenshot of the network panel in the chrome webtools after making a new live query (we're looking to see whether it is connecting over a websocket or XHR requests).

@kp625544 Can you please answer my questions in the post above, in addition to those added in this post?

[n4ll3ec]

@zwass thank you for your help. the screenshot is shown as below

[zwass]

@fl4nker it looks like Fleet is falling back to XHR requests. Unfortunately the library that provides this support was only available in JS and so there is no fallback support in fleetctl.

If you want queries to work in fleetctl you'll have to configure your LB or whatever else is fronting Fleet to support websockets.

[anelshaer]

@zwass I have the same issue when I run fleetctl on a remote machine, I'm able to see online hosts but not able to run queries. but when I run the same fleetctl query on the fleet host it's able to get results. All hosts are online in the web UI and Fleetctl also able to get hosts. fleetctl - version 2.4.0 branch: master revision: 70c6f5bd7d8a337a97f7702ee17088e74722896f build date: 2019-11-12T23:11:23Z build user: zwass go version: go1.12.8