Duplicate indicators in CRITs

[degimi]

As discussed with Alex - currently in CRITs, if an indicator exists and you run the script, in the details/Sources of the IP you can see the following:

alienvault (4): 2015-03-11

Method: trawl Reference: http://reputation.alienvault.com/reputation.data Analyst: API Created: 2015-03-11 13:20:58.188000

Method: trawl Reference: http://reputation.alienvault.com/reputation.data Analyst: API Created: 2015-03-11 13:40:17.452000

Method: trawl Reference: http://reputation.alienvault.com/reputation.data Analyst: API Created: 2015-03-11 14:17:23.745000

etc -

So, running the script everyday and the indicator is always in that feed, you will see a very long list in the "wrong place".

An idea to fix the issue could be:

1. check if the indicator exists from that source 1.1) if it is not we add it normally 1.2) if it is, and the source is different, we add the new source in CRITs and keep the information in CRITs/"sources" box example:

Method: trawl Reference: http://url.feed1/indicator Analyst: API Created: 2015-03-12 13:40:17.452000

1.3) if it is and the source is the same, we add in the CRITs/"comments" box the history in the following format http://reputation.alienvault.com/reputation.data dated 2015-03-09 http://reputation.alienvault.com/reputation.data dated 2015-03-10 http://url.feed1/indicator dated 2015-03-11 etc etc

makes sense?

[alexcpsec]

@paulpc what do you think of this? Is this doable?

[paulpc]

I liked the added sources - it keeps track of where the IP is coming from and when - a cheap way to analyze when you saw the IP and how often it appeared where it did.

Now that i'm done complaining, it's doable - it'll put more pressure on mongo and the API. I thinnk we'd have to search the API for each IP - it might be more expeditious to get a copy of everything in CRITs for the sources in combine and check for duplicates in memory before uploading. At the same time, I should have also looked to make sure the campaigns / sources exist in CRITs and if not, either prompt, or create them.

[paulpc]

@alexcpsec, here's a temporary fix for this: https://github.com/mlsecproject/combine/pull/129 I got to look at @sooshie 's branch and it has a lot of promise - once you're ready to head that direction, i would love to add my edits / ideas in there.

[alexcpsec]

@sooshie 's stuff is almost done and is going to be 0.2.0. There are still some non terminating thread oddities to fix there.

On Sat, Mar 21, 2015 at 6:16 PM, Paul Poputa-Clean [email protected] wrote:

@alexcpsec, here's a temporary fix for this: https://github.com/mlsecproject/combine/pull/129

I got to look at @sooshie 's branch and it has a lot of promise - once you're ready to head that direction, i would love to add my edits / ideas in there.

Reply to this email directly or view it on GitHub:

https://github.com/mlsecproject/combine/issues/124#issuecomment-84517698

---

This e-mail message and any files transmitted with it contain legally privileged, proprietary information, and/or confidential information, therefore, the recipient is hereby notified that any unauthorized dissemination, distribution or copying is strictly prohibited. If you have received this e-mail message inappropriately or accidentally, please notify the sender and delete it from your computer immediately.