Let's improve the website: osquery schema

[mike-j-thomas]

Goal

100% of osquery tables advertised inside the product work correctly and intuitively, or are clearly documented with caveats explaining where their behavior is not what someone might expect. Or if they're deprecated and should not be used, or if they should not be used sometimes.

How

• From: https://osquery.io/schema which is already in a programmatically parseable format because we use it in the product in the query console

• Endgame: Probably separate pages per table -- something like this: https://fleetdm.com/tables/account_policy_data

• Steps

• [ ] Wireframe what the page looks like that shows the tables consider mobile and wireframe how you get to it, and its URL McNeil: Would love to hear your theory on how to send folks to this page and its URL

Vision: Over time, add additional context to make the documentation more useful. This value-add stuff likely only lives on the fleetdm.com side, for convenience / to simplify editing the website.

Example: https://sailsjs.com/documentation/reference/request-req/req-method Versus: https://expressjs.com/en/api.html#req.method

[mike-j-thomas]

~Note to self: assign this to @DominusKelvin in the release planning session for v4.14.x~ disregard this comment.

[mike-j-thomas]

FYI: @mikermcneil I confused your request to turn the Digital Experience part of this OKR into an issue and made this issue for your entire OKR instead.

@noahtalerman this may be useful to you because it looks like you're up first to get this moving. You might want to follow suit with what I did in the description for the Digital Experience portion of this ticket.

Alternatively, we could archive this ticket and awkwardly straighten our pants as we walk away, pretending that none of this ever happened 🤦‍♂️

[mike-j-thomas]

@noahtalerman, I created this ticket when this OKR fell under Digital Experience. Please let me know if this is of any use to you. Otherwise, I think we can just close it.

[mike-j-thomas]

@noahtalerman, edit to the above. It looks like it still is listed under Digital Experience. Before we transitioned devrel, my intention was for Kathy and Kelvin to work on this documentation. Is that still possible?

[noahtalerman]

my intention was for Kathy and Kelvin to work on this documentation. Is that still possible?

@mike-j-thomas yes, this makes sense to me. Can you please make sure that this documentation task is added as a "Product" OKR in the OKRs Google sheet? https://docs.google.com/spreadsheets/d/1tFsVzX3aqfTdtnuQfYkODdkX8nDEZQ4IHxNNOIayrE0/edit#gid=0

[noahtalerman]

@xpkoala heads up, this is a public issue tracking the same goal (first checkbox in this issue's description) as fleetdm/confidential#940.

I linked fleetdm/confidential#940 to this issue.

[mike-j-thomas]

@noahtalerman, I'm just checking in about where you are with testing osquery tables? I want to start adding them to the Fleet docs as soon as possible. Thanks.

[zhumo]

@mike-j-thomas Some relevant issues for this work:

https://github.com/osquery/osquery-site/issues/189 ^ osquery has some work they want to do on their site to better document some gotchas via icons.

https://github.com/fleetdm/fleet/issues/4973 ^ More engineering/product-driven validation work on the tables. I think there have been findings from that effort which overlap with your effort here. Esp around documenting: M1 vs. intel, user context, required columns.

I will try to get on your calendar before I head out for vacation!

[mike-j-thomas]

Current webpage design ready for review https://www.figma.com/file/yLP0vJ8Ms4GbCoofLwptwS/%E2%9C%85-fleetdm.com-(current%2C-dev-ready)?node-id=4528%3A13076

@zhumo, I'm not sure where the best place to add "user context-dependent" is. Is it related to the UID col? Would it make sense to add an icon there?

[zhumo]

@mike-j-thomas

Answer to your question: yes I do believe it is related to the uid column. That said, there are two technical challenges. First is that the macos preferences table is user-dependent but does not have a uid column. I'm not sure what column that table uses instead. Second is that the uid column is not what is marked with user_dependent: true. Instead, it's the whole table that is marked, so it won't be possible (under the current data structure) to mark that individual column. I'll need to speak to eng about both...

I looked over the designs... other thoughts listed here:

1. What does it look like in the right-hand side of the query builder UI?
2. I think we'll want to link out to some article that describes these in more details. Could the icons be clickable or there be some "learn more" link or something like that?
3. I think we need more discussion on how to handle M1 vs. Intel. Strictly speaking, they are not different platforms, so I'm not sure we would elevate M1 vs. Intel by creating a new row in the drop-down here. Furthermore, each table behaves in a different weird way, so it probably isn't good enough to just say it works here but not there. We should be more specific. Yet another wrinkle is I think it'll take much longer for us to get osquery to update the json that the docs are based on, so we might need to merge our own data here or maintain it on our own. I think I need to think about this a bit more, get your view and eng's view before we move forward on how to document this.

cc: @mikermcneil from our 1/1 earlier. Notes I mentioned.

[zhumo]

Notes w/ Sharvil:

1. Looks like the preferences table uses the username column rather than uid for some reason.
2. The dependent on user context icon should be at the table level rather than the column level.
3. We should also include the "events publisher" icon at the table-level

[mike-j-thomas]

@zhumo, thanks for meeting up today. I'll get these comments and feedback in the wireframes.

[mike-j-thomas]

@zhumo, I'm sorry for keeping you waiting on this. I aim to have the designs complete by EOW.

[mike-j-thomas]

@zhumo, ~the website version (including mobile)~ Osquery schema for website and Fleet EE is ready for review. Can we grab some time early next week to go over it? Your availability is a little unclear on the Calendar. Would you mind grabbing some time from me (any time before my DigEx huddle is fine)?

https://www.figma.com/file/yLP0vJ8Ms4GbCoofLwptwS/%E2%9C%85-fleetdm.com-(current%2C-dev-ready)?node-id=4528%3A13076

~I'd also like to discuss how we can get that information into the Query Console side panel and what we might be able to omit from that view if needed.~ Ignore that. I was able to add it to Query Console without omitting any information.

[mike-j-thomas]

@zhumo I've updated the screens. Let me know if we've covered everything, and we can loop in @mikermcneil for review and @eashaw to scope out the technicalities https://www.figma.com/file/yLP0vJ8Ms4GbCoofLwptwS/%E2%9C%85-fleetdm.com-(current%2C-dev-ready)?node-id=4528%3A13076

[mike-j-thomas]

@zhumo, would you mind checking my dev notes and correcting or adding anything that I have missed?

[zhumo]

Hey Mike! Some notes:

1. re: the windows-only icon. There are also Linux- and Mac-only columns as well. We ought to accommodate those. I wonder if it makes sense to collapse those into one: have the three OS icons and call it "OS-specific"
2. re: the legend in general, I think we could clarify in a dev note whether the legend always appears or is it dynamic (as in, only certain icons appear if the table has that icon). I think static to start. [I added this in the dev notes]
3. I think we could also specify in a dev note that the Notes section should be markdown and that it only appears when there is anything there [I added this]
4. Same with Examples section. [I added this]
5. re: the note about evented tables. One way you can tell heuristically that the table is evented is to see that it ends with _events. But actually in the osquery schema json file, there is an attribute on each table evented: true/false that tells you this definitively. I think devs will prefer the latter approach. [I added this]
6. (loosely held opinion here, could discuss in group) I don't think we need to fully replicate the functionality to allow users to select the older osquery versions. In general, we set the osquery version included as part of orbit, which then gets pushed out. We could only provide a few versions or maybe we only start supporting versions going forward? Depends on how the SEO stuff works
7. What's the interaction/behavior when I search for a table? Right now, it looks like I will get a giant list of tables? I'm having some trouble imagining how the 768px and 375px screens work.

[mike-j-thomas]

@zhumo,

re: the legend in general, I think we could clarify in a dev note whether the legend always appears or is it dynamic (as in, only certain icons appear if the table has that icon). I think static to start. [I added this in the dev notes]

Unless it significantly impacts dev time, I think the legend should be dynamic. It's confusing to see, for example, "Defaults to root" if does not relate to a column in the table with the icon. Without proper context, or if a user is unaware that these icons appear in the columns of certain tables, they may infer that the whole table defaults to root. Especially a beginner.

(loosely held opinion here, could discuss in group) I don't think we need to fully replicate the functionality to allow users to select the older osquery versions. In general, we set the osquery version included as part of orbit, which then gets pushed out. We could only provide a few versions or maybe we only start supporting versions going forward? Depends on how the SEO stuff works

I agree, based on my knowledge of Orbit. If Orbit auto-updates, it makes sense that Fleet will always be running the most recent version. As you said, it depends on our SEO goals and whether we want this to be a searchable resource for vanilla osquery users.

What's the interaction/behavior when I search for a table? Right now, it looks like I will get a giant list of tables? I'm having some trouble imagining how the 768px and 375px screens work.

Thanks for bringing this up. Tbh, I hadn't considered this because I expected the search functionality to be cut from v1. Now I realize the placement for the search box is incorrect. I think it should be within the side nav and filter that list. Moving the search box would create considerable waste of screen space. So I propose all filters should be moved into the side nav (makes sense anyway as they are filtering that list) as shown here:

For 768px and 375px breakpoints:

https://user-images.githubusercontent.com/78363703/184269290-22b21109-b189-4977-b5be-ba338784737a.mov

[zhumo]

@mike-j-thomas

Unless it significantly impacts dev time, I think the legend should be dynamic.

Makes sense to me. Let's go with that.

That search functionality makes a lot more intuitive sense now, even with one-glance. Awesome!

[mike-j-thomas]

@zhumo updated designs. Please let me know when you have the JSON data ready, and I'll get the (website) build scheduled.

https://www.figma.com/file/yLP0vJ8Ms4GbCoofLwptwS/%E2%9C%85-fleetdm.com-(current%2C-dev-ready)?node-id=4528%3A13076

[zhumo]

Looks like this was done... closing!