Artifact Import pack - 0 files gives no error message

[paraknell]

Server Version

Velociraptor 0.7.0-2 built on 2023-09-10T15:23:01Z (82f2cad)

Issue

This is a two-fold problem.

1. Upload artifacts from a Zip pack gives no error messages when the internal .yaml files have syntax or other issues.
2. Documentation also does not clearly state how to run the --definitions command from cli. https://docs.velociraptor.app/knowledge_base/tips/startup_artifacts/
3. I had to find this article to determine that the artifacts list sub-command was the correct one to try, and that it further required -v --verbose flag to actually get the errors. https://docs.velociraptor.app/blog/html/2018/11/13/velociraptor_training_at_nzitf/

sudo velociraptor --definitions my_artifacts/ -v artifacts list

4. It should also be noted that on windows the upload function allows you to upload any file type, then tells you that only .zip files are accepted. It should enforce file extensions upfront, although it looks like there is sufficient code to verify the validity of the file after upload so maybe this point is mute.

Context

I was attempting to utilize some custom yara rules that would then be turned into custom artifacts so they could be run regularly/repeatedly/updated as needed. I mention this as it is not clear if this is the right way to do this, but if there is a better way then I am all ears.

I used the Generic.Detection.Yara.Glob as a baseline and then a python script to add the yara rules and then dump out the completed .yaml files. Using the above command I was finally able to discover that the python script was not properly formatting the yara rule into the final output of the final_artifact.yaml file. Something I will have to fix on my own, but in the mean time some error messages on the frontend would be quite nice!

Thanks!

[scudette]

Usually the --definitions flag is not used very often because it is very inconvenient - it only works when the server starts the first time so if you are updating artifacts all the time it is probably not the right way to go

The supported way to update artifacts is via the VQL https://docs.velociraptor.app/vql_reference/server/artifact_set/ function and that will log an error if the yaml is not correct.

Most people manage their external artifacts in another repository (e.g. on github) and periodically sync that using https://docs.velociraptor.app/artifact_references/pages/server.import.artifactexchange/ which can run on a schedule if needed.

The GUI upload method is manual and it is really only useful for when the server is blocked from egress connections out to fetch its own artifact updates so the user has to manually upload it.

Perhaps we do need to show some errors in parsing the artifacts in the GUI though

[paraknell]

The supported way to update artifacts is via the VQL https://docs.velociraptor.app/vql_reference/server/artifact_set/ function and that will log an error if the yaml is not correct.

Most people manage their external artifacts in another repository (e.g. on github) and periodically sync that using https://docs.velociraptor.app/artifact_references/pages/server.import.artifactexchange/ which can run on a schedule if needed.

parameters:
   - name: ExchangeURL
     default: https://github.com/Velocidex/velociraptor-docs/raw/gh-pages/exchange/artifact_exchange_v2.zip

Okay so if I am understanding correctly based on that VQL the repository would still be required to be in a .zip format.

As far as I can tell there isn't a way to point to a repository with yara rules in it and import those directly without some kind of modified process.

If we take this repository as an example https://github.com/Yara-Rules/rules they do not have a releases section(i.e. no zip file), the repository would need to be cloned, converted to "artifact" .yaml, zipped and stored somewhere, then pushed to the velociraptor server with a modified Server.Import.ArtifactExchange to point to a NEW repository/site URL that holds the .zip files.

Potential Pseudo Script Requirements

git clone https://github.com/Yara-Rules/rules ~/Documents/rules

python script-change-to-artifats.py ~/Documents/rules/artifiacts

zip ~/Documents/rules/artifiacts ~/Documents/my/artifact/zip/repository/rules.zip

cd ~/Documents/my/artifact/zip/repository/
git add rules.zip
git commit .....
git push .....
# or some other site to hold zip files
mv ~/Documents/my/artifact/zip/repository/rules.zip /var/www/

Then run the below example.

Example VQL modification

In the Server.Import.ArtifactExchange example VQL I would duplicate the artifact, and then only change the repository url, it should download the new .zip file, expand it and import each of the results.

parameters:
   - name: ExchangeURL
     default: https://mysite.com/rules.zip

Let me know if that doesn't make sense, but I'm honestly curious how others handle large imports for yara rules.

[scudette]

Normally people curate their own Yara rules in their own repo and prepare the artifacts eg using GitHub actions ( that's how we do it anyway )

But you can write vql to do anything including construct the artifact on the fly using the raw Yara rules. I'm sure you can build a yaml string with the output of http_client containing all the Yara rules then set the artifact into the repo.

[paraknell]

Ah I see. I will have to play around with those ideas then. Much appreciated!

[scudette]

Fixed by #3331