PyMISP
PyMISP copied to clipboard
MISP
Error: Could not add event '0' from feed XX - incompatible manifest.json file?
We are successfully generating our feeds using feed-generate.py but when we add them to a different MISP instance, the following message is presented on debug logs:
==> error.log <== 2019-12-30 17:46:11 Error: Could not add event '0' from feed XX. [InvalidArgumentException] Given event UUID '0' is invalid. Stack Trace: #0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSo cket)) #1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Arra y, Array) #2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Ar ray, '59652') #3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Arra y, '59652') #4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed() #5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform() #6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform() #7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job)) #8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5') #9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5') #10 {main}
The only file downloaded is the manifest.json and its JSON structure is correct. I can't see in it any UUID as '0', our org UUID is correct.
Could this be that the manifest.json file generated by feed-generate.py is incompatible with the latest version of MISP?
Snippet of a generated manifest.json file:
[{
"5e04405b-20f4-4f12-a6ba-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336925
},
"5e04405d-af64-4215-828c-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-25",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336927
},
"5e044060-5090-48af-a0d7-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336930
},
"5e044062-b804-4ac4-9df4-0eb6ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-27",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336932
}
}]
Thank you!
And the JSON file of an event:
{ "Event": { "extends_uuid": "", "publish_timestamp": 1577336929, "info": "Active DGA for Bamital", "threat_level_id": 1, "published": true, "analysis": 1, "date": "2019-12-25", "uuid": "5e04405d-af64-4215-828c-0af3ac110002", "timestamp": 1577336927, "Orgc": { "uuid": "5df12de7-b018-4f25-ac49-04bdac110002", "name": "My ORG" }, "Tag": [ { "colour": "#fbff30", "name": "dga" }, { "colour": "#FFC000", "name": "tlp:amber" }, { "colour": "#075200", "name": "admiralty-scale:source-reliability="b"" }, { "colour": "#0eb100", "name": "admiralty-scale:information-credibility="1"" }, { "colour": "#0088cc", "name": "misp-galaxy:botnet="Bamital"" } ], "Attribute": [ { "type": "domain", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc", "uuid": "5e04405d-0d54-4d02-b1b7-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "domain|ip", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc|175.126.123.219", "uuid": "5e04405d-2e54-429e-bc66-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "ip-dst", "comment": "", "to_ids": true, "category": "External analysis", "value": "175.126.123.219", "uuid": "5e04405d-d8a0-49b9-b35a-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "comment", "comment": "", "to_ids": true, "category": "External analysis", "value": "Bamital", "uuid": "5e04405d-dd40-400f-bd8c-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "datetime", "comment": "", "to_ids": true, "category": "Other", "value": "2019-12-25T00:00:00", "uuid": "5e04405d-312c-42f9-bbb6-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false } ] } }
I think the problem is due to the fact your manifest file is a list of dictionaries, when is should just be a dictionary.
Your manifest file should look like that:
{
"5e04405b-20f4-4f12-a6ba-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336925
},
"5e04405d-af64-4215-828c-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-25",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336927
},
"5e044060-5090-48af-a0d7-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336930
},
"5e044062-b804-4ac4-9df4-0eb6ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-27",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336932
}
}
I ran the feed generator locally and the manifest file was as expected, so I'm not sure why you have an incorrect file on your end. Can you re-try with the latest version of PyMISP from the repository>
I see. Not sure why that happened, but with the latest version of PyMISP the manifest.json looks OK. Does it look OK to you? But the error persists:
{
"5df941c1-e490-4032-a8b5-0061ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Pony",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616388
},
"5df941c4-fa90-4d05-9be5-021bac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616390
},
"5df941c7-240c-4555-ab12-0145ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616393
},
"5df941ca-a82c-4774-ab9e-005eac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616396
},
"5df941cc-979c-447e-b859-021fac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616398
}
}
The importing server still shows:
==> error.log <==
2020-01-02 17:05:42 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59659')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59659')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
I'm running Python 3.8.1 and PyMISP 2.4.119.1. The servers (both exporting and importing) run MISP v2.4.119.
I'm probably missing the point here, but now I don't see any structural difference between my manifest.json and yours. And, because my understanding of how MISP ingests feeds is very limited, I can't spot the reason for the error.
Again, sorry for the multiple messages but I'm writing as I try to debug it.
This is odd, I replaced the manifest.json file with yours and expected the importing server to produce a different error, something related to it now finding the corresponding JSON files with events or inconsistency with hashes.json. Instead, I got exactly the same error:
2020-01-02 17:32:08 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59661')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59661')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
OK, I found the problem. And it is very odd and not related to PyMISP, but to MISP itself.
The feed was hosted in an URL like this:
https://www.example.com/misp/bla:c2_address
That when MISP URL encodes, becomes:
https://www.example.com/misp/bla%3ac2_address
If I host it at:
https://www.example.com/misp/c2_address
It works! Therefore, I believe the problem is with the ':' character or its URL encoded version '%3a'. I'm not sure, but I'm relieved it works.
What is the best way to file this as a MISP bug?
Thank you very much for your assistance with this matter!
Oh wow, nice catch. Let me loop @iglocska in for that one, because I'm unsure about the way forward, and if/how we can fix it.
I still face a similiar error:
2023-01-30 18:52:42 Warning: Could not add event '5a5df804-acb5-4fd3-8c76-6982e5e1ce75' from feed 66: 1900 2023-01-30 18:52:42 Warning: Could not add event 'ca0e87d9-b850-404d-8b17-e51d2e2b717e' from feed 66: 1901 2023-01-30 18:52:42 Warning: Could not add event '83b4018e-1f45-48c4-908b-2ef8d2e1db0f' from feed 66: 1902 2023-01-30 18:52:42 Warning: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66: 1903 2023-01-30 18:52:43 Error: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66. [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'Purchase order.exe' for key 'value' Stack Trace: #0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute() #1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute() #2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1132): DboSource->execute() #3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1942): DboSource->create() #4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave() #5 /var/www/MISP/app/Model/OverCorrelatingValue.php(50): Model->save() #6 /var/www/MISP/app/Model/Correlation.php(503): OverCorrelatingValue->block() #7 /var/www/MISP/app/Model/Attribute.php(470): Correlation->afterSaveCorrelation() #8 /var/www/MISP/app/Lib/Tools/BetterCakeEventManager.php(21): Attribute->afterSave() #9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): BetterCakeEventManager->dispatch() #10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave() #11 /var/www/MISP/app/Model/Attribute.php(2518): Model->save() #12 /var/www/MISP/app/Model/MispObject.php(1134): Attribute->captureAttribute() #13 /var/www/MISP/app/Model/Event.php(3776): MispObject->captureObject() #14 /var/www/MISP/app/Model/Feed.php(1098): Event->_add() #15 /var/www/MISP/app/Model/Feed.php(693): Feed->__addEventFromFeed() #16 /var/www/MISP/app/Model/Feed.php(1182): Feed->downloadFromFeed() #17 /var/www/MISP/app/Console/Command/ServerShell.php(404): Feed->downloadFromFeedInitiator() #18 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(459): ServerShell->fetchFeed() #19 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(222): Shell->runCommand() #20 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch() #21 /var/www/MISP/app/Console/cake.php(45): ShellDispatcher::run() #22 {main}
Moreover, I face another problem probably linked to that:
The previous issue has been experienced both with simple workers and simplebackgroundjob
I don't think it is the same issue. But it is most probably not PyMISP related (please tell me if I'm wrong). Can you open an issue in the MISP repository?