PyMISP get object template from server and drop local copies from package

[chrisinmtown]

Inspired by #658 this requests a new feature: the PyMISP client MISPObject class __init__() method should query the MISP server for the object template definition (e.g., "my-internal-bad-event") instead of requiring & reading a definition.json file locally in its site-packages area. I see at least two advantages:

1. Clients could use new custom objects immediately after server administrators upload new custom object definitions.
2. Eliminating redundancy of definition files in the PyMISP package and the server will reduce maintenance effort.

To ensure performance does not suffer too much, the PyMISP client should cache object definitions.

[chrisinmtown]

The prerequisites here include at least one vital MISP server feature: search/fetch custom object (template) by name yielding a suitable format.

[Rafiot]

You do not technically need the template locally.

If you pass strict=false to the MISPObject constructor, PyMISP will not bother with the template. But (and it is an important one) you then have to provide a type and an object_relation when adding an attribute to the object:

user_defined_obj = MISPObject(name='blah', strict=False)

user_defined_obj.add_attribute('member1', type='text', value='foo')
user_defined_obj.add_attribute('member3', type='text', value='baz')
print(user_defined_obj.to_json(indent=2))

{
  "name": "blah",
  "uuid": "7fff1fca-63a8-4137-8bf2-d7db3dc6d1a9",
  "Attribute": [
    {
      "uuid": "de4f0843-881b-499c-96e4-389cc7397d5d",
      "object_relation": "member1",
      "value": "foo",
      "type": "text",
      "disable_correlation": false,
      "to_ids": false,
      "category": "Other"
    },
    {
      "uuid": "3afaec9f-b384-4154-b0f4-d2a3be660fac",
      "object_relation": "member3",
      "value": "baz",
      "type": "text",
      "disable_correlation": false,
      "to_ids": false,
      "category": "Other"
    }
  ],
  "distribution": "5",
  "sharing_group_id": "0"
}

It would in fact be nice to be able to get the templates from a MISP instance in the format of misp-objects. Can you open an issue in the MISP repository for that, please?

[chrisinmtown]

Thanks @Rafiot for the comments. I opened https://github.com/MISP/MISP/issues/6591

You have confirmed that the PyMISP client requires object template definitions locally, either as a definition.json in site-packages or embedded in the request code. Both solutions are messy because of the redundant definitions. Getting the object template definition from the server would be a superior solution.

[ticentra]

The type option is definetly better for me than keeping the templates locally! Thank you!

However, how do you add such object to MISP because with

    misp_obj = MISPObject(name='domain-ip-test-copy', strict=False)#
    misp_obj.add_attribute('domain', type='domain', value='test.domain')
    misp_obj.add_attribute('ip', type='ip-dst', value='1.1.1.1')
    print(misp_obj.to_json())
    misp.add_object(event_uuid, misp_obj)

last line throws error:

ERROR [api.py:2961 - _check_response() ] Something went wrong (403): {'saved': False, 'name': 'Could not add Object', 'message': 'Could not add Object', 'url': '/objects/add', 'errors': 'Object could not be saved.\n'}

[chrisinmtown]

I expect for this to work your server (MISP instance) must have an object template named "domain-ip-test-copy" in its filesystem like /var/www/MISP/app/files/misp-objects/objects, does it?

[ticentra]

Yep, it does:

root@[DOCKER_ID]:/var/www/MISP/app/files/misp-objects/objects/domain-ip-test-copy# ls
definition.json

[chrisinmtown]

You might post the contents of that definition.json here, my next guess is that your code's attributes and the server definition might vary? Also I recommend checking debug.log and error.log files on the server to see if it logged additional details.

[ticentra]

(I misunderstood your comment with the prev version of this reply)

domain-ip-test-copy is literally just a compy of domain-ip object template (with a different name and uuid):

{
  "attributes": {
    "domain": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "Domain name",
      "misp-attribute": "domain",
      "multiple": true,
      "ui-priority": 1
    },
    "first-seen": {
      "description": "First time the tuple has been seen",
      "disable_correlation": true,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "ip": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "IP Address",
      "misp-attribute": "ip-dst",
      "multiple": true,
      "ui-priority": 1
    },
    "last-seen": {
      "description": "Last time the tuple has been seen",
      "disable_correlation": true,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "port": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "Associated TCP port with the domain",
      "misp-attribute": "port",
      "multiple": true,
      "ui-priority": 1
    },
    "registration-date": {
      "description": "Registration date of domain",
      "disable_correlation": false,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "text": {
      "description": "A description of the tuple",
      "disable_correlation": true,
      "misp-attribute": "text",
      "recommended": false,
      "ui-priority": 1
    }
  },
  "description": "A domain and IP address seen as a tuple in a specific time frame.",
  "meta-category": "network",
  "name": "domain-ip-test-copy",
  "required": [
    "ip",
    "domain"
  ],
  "uuid": "2cc56437-97ed-4a83-b643-7848d78dc96f",
  "version": 1
}

Types checks out imo, also I've tried this with other custom template (that requires only a domain, so a simpler case) with the same result.

[chrisinmtown]

This is blocked/waiting for resolution of https://github.com/MISP/MISP/issues/6591

[chrisinmtown]

I'm happy to see progress here! https://github.com/MISP/PyMISP/commit/0d86a4339f13f9d6634ec1523a25ecae226f8340

[chrisinmtown]

MISP/MISP#6591 was merged and released about 2.4.138. I believe that added the new endpoint that yields an object template in the format expected by PyMISP. Please tell me if there's any way to help here.