Update Event doesn't add object

[franciscoGolondrina]

Hello.

I hope to add custom object to specific event using PyMISP. I'm using PyMISP 2.4.168. When i tried it, it didn't work.

I did something like below first.

test = self.misp.get_event(6, pythonify=True)
test.info = 'Object Testing '
object_test = MISPObject(name='custom-phishing', misp_objects_path_custom='files/misp-objects/objects/custom-phishing/definition.json')
object_test.add_attribute(object_relation='Category', simple_value='Hello', type='text')
test.add_object(object_test, standalone=True)
self.misp.update_event(test, pythonify=True)

The object template is correctly added to the MISP instance. Also I can add the object in the GUI but no with PYMISP

No error are given, just the event updates itself with no objects inside.

Kind regards

[Rafiot]

can you print the response of self.misp.update_event(test, pythonify=True)?

[franciscoGolondrina]

Something went wrong (403): {'saved': False, 'name': 'Could not add Object', 'message': 'Could not add Object', 'url': '/objects/add', 'errors': 'Object could not be saved.\ntemplate: No valid template found and object lacking template information. (meta-category)'}

[Rafiot]

the template you're using is lacking the meta-category, that's why it's not added.

[franciscoGolondrina]

The meta-category is added correctly.

{
  "attributes": {
    "domain": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "Domain name",
      "misp-attribute": "domain",
      "multiple": true,
      "ui-priority": 1
    },
    "first-seen": {
      "description": "First time the tuple has been seen",
      "disable_correlation": true,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "ip": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "IP Address",
      "misp-attribute": "ip-dst",
      "multiple": true,
      "ui-priority": 1
    },
    "last-seen": {
      "description": "Last time the tuple has been seen",
      "disable_correlation": true,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "port": {
      "categories": [
        "Network activity",
        "External analysis"
      ],
      "description": "Associated TCP port with the domain",
      "misp-attribute": "port",
      "multiple": true,
      "ui-priority": 1
    },
    "registration-date": {
      "description": "Registration date of domain",
      "disable_correlation": false,
      "misp-attribute": "datetime",
      "ui-priority": 0
    },
    "text": {
      "description": "A description of the tuple",
      "disable_correlation": true,
      "misp-attribute": "text",
      "recommended": false,
      "ui-priority": 1
    }
  },
  "description": "A domain and IP address seen as a tuple in a specific time frame.",
  "meta-category": "network",
  "name": "customtesting",
  "required": [
    "ip",
    "domain"
  ],
  "uuid": "28e9e822-3b2b-4a60-b1d4-8be81661b107",
  "version": 1
}

[Rafiot]

Okay, so from what I see here, there are a few things:

• name='custom-phishing' is not the name in the template (customtesting) - I think this is the core issue, you can confirm it by printing object_test.to_json(indent=2) before you add it to the event. It will most probably not have the template metadata.
• object_relation='Category' there is no object relation Category in your template - this is not a problem at MISP will just accept it, but probably not what you want. I'm guessing you want to do that: object_test.add_attribute(object_relation='text', simple_value='Hello')

[franciscoGolondrina]

My bad, wrong object template. This is the correct one.

{
  "description": "Custom phising",
  "meta-category": "misc",
  "name": "custom-phishing",
  "uuid": "f4a8d9ff-830d-44ed-9bb3-5e961d126034",
  "version": 20230221,
  "attributes": {
    "category": {
      "description": "Category",
      "disable_correlation": false,
      "misp-attribute": "text",
      "multiple": true,
      "ui-priority": 3
    },
    "date": {
      "description": "Date",
      "disable_correlation": false,
      "misp-attribute": "text",
      "multiple": true,
      "ui-priority": 3
    },
    "sender": {
      "description": "Sender",
      "misp-attribute": "text",
      "ui-priority": 3
    },
    "ip": {
      "description": "IPs",
      "disable_correlation": false,
      "misp-attribute": "text",
      "ui-priority": 2,
      "multiple": true
    }
  }
}  

[Rafiot]

Ok, can you try print(object_test.to_json(indent=2)) before the call to self.misp.update_event?

And it should be object_relation='category' (lower case, as it is in the template).

[franciscoGolondrina]

{ "name": "custom-phishing", "uuid": "73bcfda5-1e35-47db-859e-2e84cf2f63ff", "Attribute": [ { "uuid": "79e82df5-e69b-4a7a-9c43-e0e7c9e329e6", "object_relation": "category", "value": "Hello", "type": "text", "disable_correlation": false, "to_ids": false, "category": "Other" } ], "distribution": "5", "sharing_group_id": "0" }

[Rafiot]

Okay, so PyMISP cannot find the template to pre-fill the required fields.

The reason is that you're using misp_objects_path_custom='files/misp-objects/objects/custom-phishing/definition.json' which expects a path to the directory that contains the templates. It should work as expected if you pass it like that: misp_objects_path_custom='files/misp-objects/objects/'.

Otherwise, you can keep using misp_objects_path_custom but then you need to give it a python dictionary which you can get by opening the template file, and loading it with json.load.

In order to avoid such issues, and get an exception in case the template cannot be loaded, I recommend you to pass strict=True when you call MISPObject. This way, PyMISP will raise an exception telling you it is unable to get a template.

[franciscoGolondrina]

I did what you said but it gives me the next error: custom-phishing is unknown in the MISP object directory.

It sais that the template is not in the object directory but that's wrong. In the screenshot I show you that my custom object is in the folder. I tried using full path but it didn't worked.

This is the code:

test = self.misp.get_event(1, pythonify=True)
test.info = 'Object Testing '
object_test = MISPObject(name='custom-phishing', misp_objects_path_template='./files/misp-objects/objects/', strict=True)
object_test.add_attribute(object_relation='category', simple_value='Hello', type='text')
test.add_object(object_test, standalone=True)
print(object_test.to_json(indent=2))
response = self.misp.update_event(test, pythonify=True)

[Rafiot]

Just making sure:

1. You are running the script on the machine where MISP is installed?
2. If yes, the file is readable by the user you're using to run the script?
3. If not, you need to have the template file on the machine you're running the script from.

[franciscoGolondrina]

To the questions:

1. Yes, I have a local copy in the project folder.
2. I'm using root user for testing
3. The template files is on the machine and succesfully added into the MISP instance as shown in the first post.

[Rafiot]

The template being available on the MISP web interface is good, but not related to the problem.

The python script needs to open and read the file on the machine you're running the script, so the path you're currently passing will only work if you're running the script on the server, as root and from /var/www/MISP/app which is not what you're doing, right? A script running locally cannot open a file on the remote server, no matter the user you're using to run the script.

The value passed to misp_objects_path_template must be a path to a directory on the machine you're running the script that contains custom-phishing/definition.json.

[franciscoGolondrina]

I tried adding another object such as ais-info object from a local volume shared with a MISP container and with my code, the object was uploaded correctly. So I think that there is no need to run the script on the server as root and from /var/ww/MISP/app.

I just changed the object name and the object attribute and it worked correctly. The problem is with my own object template that it isn't uploading into the event. I share the same code I used but with another object. This code works correctly with other objects and not with my object.

# Works
test = self.misp.get_event(1, pythonify=True)
test.info = 'Object Testing '
object_test = MISPObject(name='ais-info', misp_objects_path_template='./files/misp-objects/objects/', strict=True)
object_test.add_attribute(object_relation='country', simple_value='Hello', type='text')
test.add_object(object_test, standalone=True)
print(object_test.to_json(indent=2))
response = self.misp.update_event(test, pythonify=True)

# Doesn't work
test = self.misp.get_event(1, pythonify=True)
test.info = 'Object Testing '
object_test = MISPObject(name='custom-phishing', misp_objects_path_template='./files/misp-objects/objects/', strict=True)
object_test.add_attribute(object_relation='category', simple_value='Hello', type='text')
test.add_object(object_test, standalone=True)
print(object_test.to_json(indent=2))
response = self.misp.update_event(test, pythonify=True)
print(response)

[Rafiot]

No, there is no need to run the script from the server, but the script needs to be able to access the file.

Can you run ls ./files/misp-objects/objects/ from the same directory you run your script?

[franciscoGolondrina]

As you can see, I put the wrong path of MISP objects but why is ais-info correctly added to the event?

Also I added the new path to the code and it isn't still founding the object template. The new code is this:

test = self.misp.get_event(1, pythonify=True)
test.info = 'Object Testing '
object_test = MISPObject(name='custom-phishing', misp_objects_path_template='./misp-core/files/misp-objects/objects/', strict=True)
object_test.add_attribute(object_relation='category', simple_value='Hello', type='text')
test.add_object(object_test, standalone=True)
print(object_test.to_json(indent=2))
response = self.misp.update_event(test, pythonify=True)

[Rafiot]

The templates in misp-objects are bundled-in the package (see path), so you can get them by name without manually giving a path to the MISPObject class (that's the case for ais-info).

What is the error message you're getting? And are you running the python script from /home/ubuntu/docker-misp?

Also, as I was just making sure: the code is tested here.

[franciscoGolondrina]

The error message I have is custom-phishing is unknown in the MISP object directory. I'm running the script from /home/ubuntu/docker-misp.

[Rafiot]

Got it. You started to use misp_objects_path_template, and I missed that. This parameter is not correct. You need to use misp_objects_path_custom instead.

And just to make sure we don't have another issue is cat ./misp-core/files/misp-objects/objects/custom-phishing/definition.json returning the template?

[franciscoGolondrina]

Finally the error is solved using misp_objects_path_custom instead of misp_objects_path_template. You were right!!!! Thank you so much for helping :) In the PyMISP documentation there are no evidences of these parameters, it could be great if you add this.

Kind regards.