fleet
fleet copied to clipboard
fleetdm
🎸 Fleet to set/important YARA rules like osquery query packs
| User story |
|---|
| As a detection & response engineers, |
| I want to deploy YARA rules to agents using the Fleet server |
| so don't have to write rules to disk (too large of scope and too slow) or host rules on a separate webserver (can't be private). |
Scope: Any engineering (including outside detection & response team) can read these rules because they live in a monorepo. If an engineer's account get's compromised then they can read these rules.
Too slow: It takes months to get these rules reviewed.
Can't be private: Needs to be private.
Ideal workflow: Customer would have a repo of YARA rules that only detection & response team can access. Deploy these per team (production servers and workstations).
Some tables are blocked by not having YARA
Future: How to displace CrowdStrike? Event monitoring. osquery evented tables aren't as good and performant as CrowdStrike.
Problem
Background:
- Currently not using YARA scanning at all today
- One option - put the rules on disk, it would take forever. If there was an incident and they needed to add new rules, that timeline wouldn’t work. Not a tenable option.
- Another option - pull from a web server, but then it would need to be publicly accessible. Then attackers could then see what their rules are.
- DART team currently has a task to build a YARA scanning capability. Osquery provides this capability but the methods to implement are not ideal for our environment
Potential solutions
- Build the feature for Fleet to be given a set of YARA rules like Osquery query packs. Next, build the feature for Osquery to pull those Osquery rules from the TLS server.
- Where would this live in Fleet?
- Same as query packs
- Would be able to be configured as ‘YARA events’ scheduled to run at a recurring configurable interval of frequency
- Benefit to them: YARA rules would be the method of distributing out the rule and detecting the presence of new malware signatures out in their environment
- Crowdstrike doesn’t support YARA rules today
- Would make it easier for other DART teams to use this feature (on a moments notice, be able to write a rule for new
Feature fest: We have a yara table in Fleet: https://fleetdm.com/tables/yara
Does that address the need?
The yara & yara_events tables do not have the capabilities to access sigrules from a remote server securely.
@noahtalerman let me know if you or design team would like to chat with the customer during or after this air guitar. thanks!
Hey @zayhanlon! Would love some help setting up a call w/ the customer.
Containers supported by YARA (Specifically Kubernetes): Brock: Need context → Adding namespace column by PID would allow you to use YARA rules in containers kubequery in the repo may be helpful May be a blocker since they don’t have direct Kubernetes API Brock: Could pass along to infra team to use Ben: Potentially, could be hard though Concern was if they could use osquery to run YARA rules in containers that are Docker → Now use Kubernetes so want that same ability for Kube osquery does not support containerd → this would resolve the issue TODO Brock: Bring this to eng to discuss Found article about osquery support for containerd https://developer.ibm.com/articles/monitoring-containers-osquery/
Hey @noahtalerman I did reach out to IBM on their GitHub to see if they had open-sourced anything in relation to this. Got no response. That said, I think we could duplicate what was built based on the article.
Hey @zayhanlon, I updated this air guitar to the user story format and moved our original issue description below.
Bringing this one back to feature fest to weigh whether we'll commit to solving this one next sprint or in a later sprint.
Original issue description:
Ideal workflow: Customer would have a repo of YARA rules that only detection & response team can access. Deploy these per team (production servers and workstations).
Some tables are blocked by not having YARA
Future: How to displace CrowdStrike? Event monitoring. osquery evented tables aren't as good and performant as CrowdStrike.
Problem
Background:
- Currently not using YARA scanning at all today
- One option - put the rules on disk, it would take forever. If there was an incident and they needed to add new rules, that timeline wouldn’t work. Not a tenable option.
- Another option - pull from a web server, but then it would need to be publicly accessible. Then attackers could then see what their rules are.
- DART team currently has a task to build a YARA scanning capability. Osquery provides this capability but the methods to implement are not ideal for our environment
Potential solutions
- Build the feature for Fleet to be given a set of YARA rules like Osquery query packs. Next, build the feature for Osquery to pull those Osquery rules from the TLS server.
- Where would this live in Fleet?
- Same as query packs
- Would be able to be configured as ‘YARA events’ scheduled to run at a recurring configurable interval of frequency
- Benefit to them: YARA rules would be the method of distributing out the rule and detecting the presence of new malware signatures out in their environment
- Crowdstrike doesn’t support YARA rules today
- Would make it easier for other DART teams to use this feature (on a moments notice, be able to write a rule for new
Hey @zwass, I'm passing this story to you and tagged you as the product designer.
Please bring this story + your proposed changes to the design review ritual so that we can provide a second pair of eyes on changes and give feedback.
Thanks!
cc @zayhanlon
Hey @zayhanlon because this is being worked on by customer success I think it makes sense to move this issue to the #g-customer-success board. I moved it.
When @zwass brings changes through design review and we decide that it's ready to estimate, I think at that point we add the issue back to the drafting board (:product) so that we can bring it through a product group's estimation session.
Please let me know if you disagree!
sounds good! i was tracking it separately, but we can follow your guidance. this will be zach's next issue after he's back from leave.
Osquery support
I put up a PR to allow osquery to authenticate yara requests: https://github.com/osquery/osquery/pull/8437. I'm working to get this into the 5.14 release this week. With this flag enabled, osquery will send the node key when retrieving yara rules, which will allow the Fleet server to authenticate the request before responding.
UX in Fleet
A very minimal PoC has been implemented so far and this UX will be fully implemented once it has been reviewed.
Configure agent options:
config:
options:
# other options ommitted
yara_sigurl_authenticate: true # "on" switch for using YARA rules in Fleet
yara:
signature_urls:
- https://localhost:8080/api/osquery/yara/.* # (Fleet server URL) Also required for using YARA rules in Fleet
Configure yara URLS in default.yml:
org_settings:
yara_rules:
- path: ../lib/test1.yar
- path: ../lib/test2.yar
Note: The filenames must be unique, as the filename becomes the last part of the path used in the sigurl. Fleet will throw an error if the user attempts to use multiple files with the same name.
(For now the yara rules will be configured at the global level. For targeting by team, target the live or scheduled queries to the appropriate team.)
Reference the rules in queries (live or scheduled):
SELECT * FROM yara WHERE path="/bin/ls" AND sigurl='https://localhost:8080/api/osquery/yara/test1.yar'
Osquery will then send an authenticated request. If Fleet considers the node key valid it will respond with the appropriate rule file.
@zwass looks good! When you get the chance, can you please open a PR to reference docs branch w/ the proposed YAML file changes? Here's the YAML reference docs: https://fleetdm.com/docs/configuration/yaml-files
Also, can you please include the proposed API changes to /config endpoints (I think PATCH and GET) in that PR?
Hey @sharon-fdm, @zwass is helping out the Endpoint ops team by building this user story.
I added this story to the release board so that we can treat it like any other story (in progress => in review => ready for QA => ready for release).
cc @lukeheath @zayhanlon
As a user writing a query, if the YARA rule doesn't exist, the query will return no results and a helpful error message will be logged in the osquery status logs.
As a user hosting YARA rules outside of Fleet, to migrate to YARA rules with Fleet, they would have to add yara_sigurl_authenticate and signature_urls to their agent options. This would turn on the feature.
If they forget to add the right URL to signature_urls, then they will run into the no results + error in status logs (scenario above).
If they forget to add the yara_sigurl_authenticate, the user will get a 403 error.
@noahtalerman: validation for signature URLs?
Hey @zwass, I think this^ would be a good thing to document in the guide.
@noahtalerman, @lukeheath, this has a long history and is currently on our release board. Should we give it a P2 / reprioritize / other ?
@sharon-fdm It looks like @zwass is owning this story and tracking on the customer success board. Noah left it on the endpoint ops board so that it could go through the normal QA and release process. But, nothing for you to track on this until it's ready for QA.
Hey @zwass can you please open a PR against the reference docs branch (assuming this story will be shipped in 4.59) w/ the proposed YAML file and REST API changes?
YAML reference docs: https://fleetdm.com/docs/configuration/yaml-files
API reference docs: https://fleetdm.com/docs/rest-api/rest-api
YARA rules deployed, Silent as a cloud's soft drift, Private, swift, and sure.
Thanks @zwass!
YAML reference docs: https://fleetdm.com/docs/configuration/yaml-files
API reference docs: https://fleetdm.com/docs/rest-api/rest-api
Re-opening this one. Just realized we never got to the reference doc updates.
Can you please open a PR to update the YAML and API reference docs?
@zwass also, did this get QA'd by one of our QA engineers? If not, can you please add it to the QA column on the #g-endpoint ops board and tag Sharon to let him know?
cc @zayhanlon
I reviewed the PR and this was shipped in 4.60.0. If we missed QA it was probably because it has :product and doesn't have the :release (and doesn't have a Milestone set).
If we missed QA it was probably because it has :product and doesn't have the :release (and doesn't have a Milestone set).
I added :product yesterday (after we shipped). Sounds like this one missed QA. We want to make sure the specs in the "Product" section are filled out so QA knows the definition of done.
- [ ] YAML changes: TODO: Specify changes in the YAML files doc page as a PR to the reference docs release branch. Put "No changes" if there are no changes necessary.
- [ ] REST API changes: TODO: Specify changes in the the REST API doc page as a PR to reference docs release branch. Put "No changes" if there are no changes necessary. Move this item to the engineering list below if engineering will design the API changes.
- [ ] Permissions changes: TODO: Specify changes in the Manage access doc page as a PR to the reference docs release branch. If doc changes aren't necessary, explicitly mention no changes to the doc page. Put "No changes" if there are no permissions changes.
- [ ] Changes to paid features or tiers: TODO: Specify changes in pricing-features-table.yml as a PR to reference docs release branch. Specify "Fleet Free" and/or "Fleet Premium" if there are no changes to the pricing page necessary.
@zwass can you please add PRs/specs for these remaining TODOs.
(I think I had removed them we were working on this story. Whoops from me)
When those are spec'd can you please add :release and #g-endpoint-ops labels and ping Sharon that this one needs QA?
cc @zayhanlon
@lucasmrod @noahtalerman -- we merged these docs awhile ago https://github.com/fleetdm/fleet/pull/24015/files but i can't figure out where it is or where it ended up on the website. did we put it in the wrong place?
@lucasmrod @noahtalerman -- we merged these docs awhile ago https://github.com/fleetdm/fleet/pull/24015/files but i can't figure out where it is or where it ended up on the website. did we put it in the wrong place?
Ah, it was merged to the docs-v4.61.0 branch so the docs will be released when 4.61.0 is released.
Ah, seems like I may have been confused about when it was going out. Maybe should merge it to main now if it shipped to 4.60? Apologies that I'm not on the ball with these process things.
@sharon-fdm i dont think this went through formal qa last sprint, so we're circling it back for qa in case it was missed per noah's comments above. can you check this out and also review if the docs are in the right place? @xpkoala fyi