The template (vulnerability) doesn't have the object_relation (vulnerable_configuration) you're trying to add.

[cudeso]

When doing a search I get the error message The template (vulnerability) doesn't have the object_relation (vulnerable_configuration) you're trying to add. (> https://github.com/MISP/PyMISP/blob/main/pymisp/mispevent.py#L1032)

events_object = misp.search(object_name="vulnerability", date_from="2022-01-01", pythonify=True, limit=250)

The 'vulnerable' object definition is version 8, both in MISP and PyMISP (lib/python3.8/site-packages/pymisp/data/misp-objects/objects/vulnerability/). It contains vulnerable-configuration instead of vulnerable_configuration.

example event UUID 30839540-715f-46fa-9fe3-a3b0b441d01e

[Rafiot]

That's annoying. I think the only realistic solution is to add another object_relation called vulnerable_configuration because if we only rename it in the template, the objects created with vulnerable-configuration will fail.

The name of the object_relation was changed in the version 6 of the template, the 26th of April 2020.

@adulau what do you think?

Edit: nevermind, I'll fix that, we don't need that check in this case.

[Rafiot]

Okay, this is even more confusing: PyMISP should already support the objects coming from old templates. Then I look at the event 30839540-715f-46fa-9fe3-a3b0b441d01e, I cannot find any object_relation called vulnerable_configuration or vulnerable-configuration. Has it been edited on an other instance or something like that?

In theory, a template is marked as unknown and all the checks are skipped unless we have a matching name, template UUID and version. In your case, the version should be different.

[cudeso]

Okay, this is even more confusing: PyMISP should already support the objects coming from old templates. Then I look at the event 30839540-715f-46fa-9fe3-a3b0b441d01e, I cannot find any object_relation called vulnerable_configuration or vulnerable-configuration. Has it been edited on an other instance or something like that?

I should really learn to copy / paste the proper output :( apologies. It concerns event 3410ad13-ef34-48c9-bc6f-b1b111a30e06 ; that one has a 'vulnerable_configuration' attribute (object UUID f793c30c-02de-4e84-8494-e06fc3013958).

[cudeso]

If I'm looking at this correctly, there's no bug, it's only the source event that used a wrong object-template?

[Rafiot]

So yes and no. This module is pushing objects with the old format. PyMISP logs it as a warning but doesn't block it (it is in line with how MISP handles objects by default). I'm considering changing the way PyMISP handles that case and actually raises an exception, because that sounds like a bug in the code using PyMISP and the dev should be informed asap. What do you think?

PyMISP should be able to handle that case anyway, so I'm working on a fix right now, and the MISP Module that generates the data also needs to be fixed (on it after the PyMISP patch).

[Rafiot]

..... now I'm the idiot. You're talking of a warning message, not an actual exception.

This warning message is there exactly for that reason. It handles broken-ish objects but still allows you to load them in PyMISP. So the actual bug is only in the module. Working on a fix now.

[Rafiot]

Fix for the module: https://github.com/MISP/misp-modules/commit/81ec6fe415cef7e6d55c6e03c37551c37f696753

[github-germ]

During my coffee wake-up, I just noticed a similar WARNING in my morning log file review. Might this have a similar root cause and fix? We're on 2.4.159.

2022-07-29T10:17:04Z WARNING The template (ip-api-address) doesn't have \
the object_relation (ip-dst) you're trying to add.

This also occurs when our client calls pymisp.search. I tried setting a breakpoint in the debugger to where mispevent.add_attribute logs that, but didn't hit it?

Note: The search is retrieving based on an eventinfo that does not use that MISPObject

Excerpt from pymisp.mispevent.add_attribute

        if self._known_template and self._definition:
            if object_relation in self._definition['attributes']:
                attribute = MISPObjectAttribute(self._definition['attributes'][object_relation])
            else:
                # Woopsie, this object_relation is unknown, no sane defaults for you.
                logger.warning("The template ({}) doesn't have the object_relation ({}) you're trying to add.".format(self.name, object_relation))
                attribute = MISPObjectAttribute({})
        else:
            attribute = MISPObjectAttribute({})

[Rafiot]

Yep, that's the same thing. The object is technically valid, but the thing generating it is not using the template. Do you know what created it?

[github-germ]

Puzzled... The MISPEvent retrieved via the pymisp.search, which triggers that WARNING, contains only MISPAttribute.

[Rafiot]

That makes sense, you won't have the source unless the tool/person who created it encoded the information in the comments for example. Is it in an event you received from a 3rd party? Or did you create it yourself? If you can share the UUID of the event, and I have it on my MISP instance, I can investigate.

If it is something created internally on your side, you need to dig up which script is creating ip-api-address objects.

[github-germ]

We created the event with our own app sourced from data from a premium feed provider. Once again there are no objects in that event.

Are you saying that either PyMISP or MISP for some reason (unknown to me) are examining an object that has nothing to do with this event? Is is possible that in constructing the MISPEvent to pass back to the client that object is being visited via a correlations linkage?

[Rafiot]

The excerpt you pasted above comes from MISPObject.add_attribute -> https://github.com/MISP/PyMISP/blob/58976dc35c33c9fc9c9abfbc9d8872c32ee01bef/pymisp/mispevent.py#L1027, so there is an object somewhere in that event.

Your search method is called with pythonify=True and it creates a MISPObject, even if you don't use it at any point.

[github-germ]

Aha, it's from an enrichment getting included into the event. Tricky...

Thanks!

And indeed, there's no ip-dst defined

$ grep ip-dst python3.8/site-packages/pymisp/data/misp-objects/objects/ip-api-address/definition.json
$ 

mysql> select * from attributes where event_id=288363 and object_relation='ip-dst' \G
*************************** 1. row ***************************
                 id: 363816030
           event_id: 288363
          object_id: 15570168
    object_relation: ip-dst
           category: Network activity
               type: ip-dst
             value1: W.X.Y.Z
             value2:
             to_ids: 1
               uuid: 35db2a9f-b604-464b-9b69-035ca09a95f4
          timestamp: 1657828125
       distribution: 5
   sharing_group_id: 0
            comment: W.X.Y.Z: Enriched via the shodan module
            deleted: 0
disable_correlation: 0
         first_seen: NULL
          last_seen: NULL
1 rows in set (0.01 sec)

mysql>

[Rafiot]

Right, so it is probably somewhere around this line: https://github.com/MISP/misp-modules/blob/main/misp_modules/modules/expansion/shodan.py#L73

[github-germ]

Is it "interesting" that the module created the object with a ip-dst attribute type which is not in the definition.json?

[Rafiot]

Yes, an object can contain anything. It is not good practice, but it is possible. The reason is that a specific instance may not have the template you're using, or not the same version.

[packet-rat]

I think I'm tracking this correctly(???):

Why not simply 'fix' the definition.json? [Either directly or have the author do so]

[Rafiot]

@packet-rat I'm not sure how the shodan module is supposed to work, so it is a bit hard to say if it is a bug in the module, or a missing entry in the template.

@chrisr3d: could you have a look? I think you're the last one who worked on the module.

[chrisr3d]

The ip-dst (or ip-src) attribute is actually the input attribute, but I probably failed a copy paste here

We do not need to add this attribute to the ip-api-address object nor to the Event (since it is already added when the class is initiated, in order to avoid references to the input attribute to be skipped), so I removed it with https://github.com/MISP/misp-modules/commit/71d8745b9137fa7e2cfacc8ac140613123f25581, it should be working as expected now

[github-germ]

@chrisr3d -- Thanks!