import_collection/Server.Utils.ImportCollection does not import results for custom artifacts not defined on the server

[chris-coalitionir]

summary: import_collection does not import custom artifact data if you're importing into a server that doesn't have the custom artifact defined/created; instead, it treats the Custom.My.Artifact.json file as an upload.

environment: velociraptor v0.6.2, linux (ubuntu 20.x)

steps to reproduce:

1. Create an custom artifact
2. Create an offline collector with that artifact and run it on a target system
3. Attempt to import the result on a different server that doesn't have that custom artifact defined
4. You'll see it upload instead of import, in contrast to an adjacent built-in artifact that works as expected:

analysis: looks like this traces back to: https://github.com/Velocidex/velociraptor/blob/44d258515b595b90596ed2e0dda821a08dfd60cd/services/repository/repository.go#L354

... where the custom artifact name is not in that Data array (which of course why would it be 🙂).

fix: I think the expected behavior if I'm importing a collector would be for it to import even if I don't have the requisite artifact "installed" ... that doesn't seem like a prereq for analyzing the data. If that's not feasible for some reason, or if I'm misunderstanding the root cause, I'd prefer an non-terminating error (letting the rest of them import). The "silently import as if it's a file upload" seems worst of all, especially for json files at the root of the zip matching the namespaced artifact format 😃 .

We're among that subset of users where ergonomics of offline collector uploads matters, perhaps a minority. I'll submit a PR for allowing a CSV list of imports for the built-in Server.Utils.ImportCollection artifact, but would love the ability to upload the zips via the UI. That's for a different issue! Thanks for everything you do.

[scudette]

When we import the collection we create "virtual" client and virtual collections for it. We can not create a collection for an artifact that we dont know about its definition, since there are many assumptions around the place with that.

We use the definitions to recognise if the zip file contains an artifact or just random data.

Maybe we can import the artifact with some kind of generic name? but this can be confusing.

Uploading the zip via the UI is problematic because we need to support uploading of huge files (currently uploads are done via http POST). It doesnt really buy all that much because there needs to be some way to transfer large data to the server anyway. It might be possible to run a something like webdav server on the Velociraptor GUI and just use the Velociraptor reverse proxy to integrate seemlessly.

[scudette]

I think the whole issue stems from us failing to put enough information in the zip file about collection in general. I would like to add some metadata to the collection that helps the importer know what to do:

1. A list of artifacts we collected
2. Collection time, hostname etc -> basically this can be used to create a better "virtual" client and a better "virtual" flow.
3. Potentially include the source of any custom VQL or artifacts that were used - even just as a way to provide provenance of what took place.

[scudette]

This should now be fixed