ExpandedPyMISP.search_index(): published attribute not working

[LaumiH]

I tried to get specific events that have been published, but it does not matter if I put published=True or published=False as an attribute in the ExpandedPyMISP.search method.

The resultset is always the same and gives both published and unpublished events.

Code:

from pymisp import ExpandedPyMISP
from keys import url, key, verifycert
misp = ExpandedPyMISP(url, key, verifycert)
result = misp.search_index(published=True, date_from=<...>, date_to=<...>, eventinfo='NRD', pythonify=True)

[Rafiot]

Are you sure? because this part of the code is tested: https://github.com/MISP/PyMISP/blob/master/tests/testlive_comprehensive.py#L672

Can you show me the outputs in both cases?

[kovacsbalu]

Looks strange for me also:

misp = ExpandedPyMISP(MISP_URL, MISP_KEY, False)
result = misp.search_index(published=True, eventinfo='test', pythonify=True)
for x in result:
    print(x.id, x.info, x.published)

1764 test True
1762 test False

@Rafiot you use misp.search() but in this issue there is misp.search_index() misp.search(published=True shows only 1764 in my example

[LaumiH]

First of all, sorry for not responding, I was sick at home. And also sorry for providing so little information, it is my first bug. I changed the issue title to the correct function, search_index(). The MISP version I am using is 2.4.112, I cannot test on newer versions as it is a company MISP.

The code @kovacsbalu provided would be exactly the sample code and output that shows the problem. The published attribute just has no effect on the events in the response.

My code that shows how published=True is not working is the following:

result = misp.search_index(published=True, date_from='2019-09-15', date_to='2019-09-26', eventinfo='NRD', pythonify=True)
for event in result:
    print(event['id'], event['publish_timestamp'].strftime('%Y-%m-%d'), event['published'])

This gives (excerpt):

193624 1970-01-01 False
193623 1970-01-01 False
193618 2019-09-24 True
193617 2019-09-24 True
193609 1970-01-01 False
193608 1970-01-01 False
193607 1970-01-01 False

and so on.

Also when I compare the numbers, there are 394 "published" events, and 391 "unpublished" events in that timespan out of total 394 events, though when I filter for publish_timestamps unequal epoch 0, there really are only 3 published events, which is also the number I get when I filter in the browser GUI.

I noticed that published=False however gives only the unpublished events, so maybe only the published=True option is not working correctly.

[chrisinmtown]

I can reproduce this bug with PyMISP/MISP version(s) 2.4.133, that calling search_index with published=False works as expected but published=True does NOT. Calling misp.search_index() against my instance with eventinfo='Threat' (just to reduce the number of results) and published=True yields 237 events, some with published=True and others with published=False. Removing the published= parameter entirely from the call to search_index() still yields 237 results. Calling with published=False yields 36 events. Ouch! Hope this is a quick & easy fix.

[chrisinmtown]

Please tell me if this code in EventsController.php, about line 289, looks right to you, bcos it looks wrong to me. I don't see how the value for key 'published' could ever be 2. This could possibly be a copy-paste error, maybe code for case 'hasproposal' mistakenly was put into case 'published'?

                    case 'published':
                        if ($v == 2) {
                            continue 2;
                        }
                        $this->paginate['conditions']['AND'][] = array('Event.' . substr($k, 6) . ' =' => $v);
                        break;
                    case 'hasproposal':
                        if ($v == 2) {
                            continue 2;
                        }

[Rafiot]

@MISP/developers poke?

[chrisinmtown]

@Rafiot should I open an issue in MISP, since this seems to be a server-side problem, not a Python client problem?

[Rafiot]

Yes, go for it please.

[chrisinmtown]

Issue written at MISP: https://github.com/MISP/MISP/issues/6642

[chrisinmtown]

Here's the explanation from @mokaddem: https://my-misp:12345/events/index/sort:published/direction:desc/searchpublished:0 yields only unpublished events and https://.../searchpublished:1 yields only published events. Next thing to check is if False and True Boolean values given to the PyMISP client are being properly transformed into zero and one for sending to the server.

[Rafiot]

There is a bugfix coming on MISP side, it should accept true/false too (at least in the POST blob, which is what PyMISP is doing)

[chrisinmtown]

Thanks @Rafiot I should have expressed myself better in https://github.com/MISP/MISP/issues/6642 to say, accept True in the POST blob. Did you open a different issue?