Determine the data type before uploading to the observable

[dumprop]

benkowcc.py:

Aug 23 07:04:43 yeti celery[1466]: [2022-08-23 07:04:43,474: ERROR/ForkPoolWorker-4] 'morihv04.top' is not a valid 'Ip'
Aug 23 07:04:43 yeti celery[1466]: [2022-08-23 07:04:43,474: ERROR/ForkPoolWorker-4] exception
Aug 23 07:04:43 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:04:43 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/benkowcc.py", line 52, in analyze
Aug 23 07:04:43 yeti celery[1466]:     ip_obs = Ip.get_or_create(value=ip)
Aug 23 07:04:43 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:04:43 yeti celery[1466]:     obj.clean()
Aug 23 07:04:43 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:04:43 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:04:43 yeti celery[1466]: core.errors.ObservableValidationError: 'morihv04.top' is not a valid 'Ip'

cruzit.py:

Aug 11 10:04:24 yeti celery[17879]: [2022-08-11 10:04:24,580: ERROR/ForkPoolWorker-6] '' is not a valid 'Ip'
Aug 11 10:04:24 yeti celery[17879]: [2022-08-11 10:04:24,580: ERROR/ForkPoolWorker-6] exception
Aug 11 10:04:24 yeti celery[17879]: Traceback (most recent call last):
Aug 11 10:04:24 yeti celery[17879]:   File "/opt/yeti/plugins/feeds/public/cruzit.py", line 38, in analyze
Aug 11 10:04:24 yeti celery[17879]:     obs = Ip.get_or_create(value=ip)
Aug 11 10:04:24 yeti celery[17879]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 11 10:04:24 yeti celery[17879]:     obj.clean()
Aug 11 10:04:24 yeti celery[17879]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 11 10:04:24 yeti celery[17879]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 11 10:04:24 yeti celery[17879]: core.errors.ObservableValidationError: '' is not a valid 'Ip'

phishing_database.py:

Aug 23 07:07:54 yeti celery[1466]: [2022-08-23 07:07:54,826: ERROR/ForkPoolWorker-4] 'https://thefurthersupdate.shop/qt/us/ipad/fa1/tt/index2.html' is not a valid 'Url'
Aug 23 07:07:54 yeti celery[1466]: [2022-08-23 07:07:54,826: ERROR/ForkPoolWorker-4] exception
Aug 23 07:07:54 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:07:54 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/phishing_database.py", line 37, in analyze
Aug 23 07:07:54 yeti celery[1466]:     url = Url.get_or_create(value=url)
Aug 23 07:07:54 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:07:54 yeti celery[1466]:     obj.clean()
Aug 23 07:07:54 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:07:54 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:07:54 yeti celery[1466]: core.errors.ObservableValidationError: 'https://thefurthersupdate.shop/qt/us/ipad/fa1/tt/index2.html' is not a valid 'Url'
Aug 23 07:07:55 yeti celery[1466]: [2022-08-23 07:07:55,177: ERROR/ForkPoolWorker-4] 'https://everythinggoesales.shop/qt/us/ipad/fa1/tt/index2.html' is not a valid 'Url'
Aug 23 07:07:55 yeti celery[1466]: [2022-08-23 07:07:55,177: ERROR/ForkPoolWorker-4] exception
Aug 23 07:07:55 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:07:55 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/phishing_database.py", line 37, in analyze
Aug 23 07:07:55 yeti celery[1466]:     url = Url.get_or_create(value=url)
Aug 23 07:07:55 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:07:55 yeti celery[1466]:     obj.clean()
Aug 23 07:07:55 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:07:55 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:07:55 yeti celery[1466]: core.errors.ObservableValidationError: 'https://everythinggoesales.shop/qt/us/ipad/fa1/tt/index2.html' is not a valid 'Url'
Aug 23 07:07:58 yeti celery[1466]: [2022-08-23 07:07:58,413: ERROR/ForkPoolWorker-4] 'http://h3htvz3q3.seufacilitador.com./#.aHR0cHM6Ly9zMy5hbWF6b25hd3MuY29tL2FwcGZvcmVzdF91Zi9mMTY2MDU0Mjc3OTA4M3g3ODIwMzUyNDQ2NDUyMDU5MDAvb3V0c3RhbmAug 23 07:07:58 yeti celery[1466]: [2022-08-23 07:07:58,413: ERROR/ForkPoolWorker-4] exception
Aug 23 07:07:58 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:07:58 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/phishing_database.py", line 37, in analyze
Aug 23 07:07:58 yeti celery[1466]:     url = Url.get_or_create(value=url)
Aug 23 07:07:58 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:07:58 yeti celery[1466]:     obj.clean()
Aug 23 07:07:58 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:07:58 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:07:58 yeti celery[1466]: core.errors.ObservableValidationError: 'http://h3htvz3q3.seufacilitador.com./#.aHR0cHM6Ly9zMy5hbWF6b25hd3MuY29tL2FwcGZvcmVzdF91Zi9mMTY2MDU0Mjc3OTA4M3g3ODIwMzUyNDQ2NDUyMDU5MDAvb3V0c3RhbmRpbmctd29zaAug 23 07:07:59 yeti celery[1466]: [2022-08-23 07:07:59,901: ERROR/ForkPoolWorker-4] 'https://thetrusthappens.shop/qt/us/13/fa4/tt/index1.html' is not a valid 'Url'
Aug 23 07:07:59 yeti celery[1466]: [2022-08-23 07:07:59,901: ERROR/ForkPoolWorker-4] exception
Aug 23 07:07:59 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:07:59 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/phishing_database.py", line 37, in analyze
Aug 23 07:07:59 yeti celery[1466]:     url = Url.get_or_create(value=url)
Aug 23 07:07:59 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:07:59 yeti celery[1466]:     obj.clean()
Aug 23 07:07:59 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:07:59 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:07:59 yeti celery[1466]: core.errors.ObservableValidationError: 'https://thetrusthappens.shop/qt/us/13/fa4/tt/index1.html' is not a valid 'Url'

threatview_c2.py:

Aug 23 07:08:47 yeti celery[1466]: [2022-08-23 07:08:47,462: ERROR/ForkPoolWorker-4] exception
Aug 23 07:08:47 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:08:47 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/threatview_c2.py", line 22, in update
Aug 23 07:08:47 yeti celery[1466]:     for ipaddr in self.update_lines():
Aug 23 07:08:47 yeti celery[1466]:   File "/opt/yeti/core/feed.py", line 448, in update_lines
Aug 23 07:08:47 yeti celery[1466]:     r = self._make_request(headers=headers, auth=auth, verify=verify)       
Aug 23 07:08:47 yeti celery[1466]:   File "/opt/yeti/core/feed.py", line 384, in _make_request
Aug 23 07:08:47 yeti celery[1466]:     last_mod, r.status_code
Aug 23 07:08:47 yeti celery[1466]: core.errors.GenericYetiInfo: Last modified date: 2022-08-11 15:56:50+00:00 returns code: 200

urlhaus.py:

Aug 23 07:11:44 yeti celery[1466]: [2022-08-23 07:11:44,210: ERROR/ForkPoolWorker-6] 'http://220819215332490.xaf.tic53.shop/f/fszmix0819.exe' is not a valid 'Url'
Aug 23 07:11:44 yeti celery[1466]: [2022-08-23 07:11:44,211: ERROR/ForkPoolWorker-6] exception
Aug 23 07:11:44 yeti celery[1466]: Traceback (most recent call last):
Aug 23 07:11:44 yeti celery[1466]:   File "/opt/yeti/plugins/feeds/public/urlhaus.py", line 68, in analyze
Aug 23 07:11:44 yeti celery[1466]:     url_obs = Url.get_or_create(value=url)
Aug 23 07:11:44 yeti celery[1466]:   File "/opt/yeti/core/database.py", line 76, in get_or_create
Aug 23 07:11:44 yeti celery[1466]:     obj.clean()
Aug 23 07:11:44 yeti celery[1466]:   File "/opt/yeti/core/observables/observable.py", line 224, in clean
Aug 23 07:11:44 yeti celery[1466]:     "'{}' is not a valid '{}'".format(self.value, self.__class__.__name__)
Aug 23 07:11:44 yeti celery[1466]: core.errors.ObservableValidationError: 'http://220819215332490.xaf.tic53.shop/f/fszmix0819.exe' is not a valid 'Url'

[sebdraven]

For URL the problem is the validation with our own regex https://github.com/yeti-platform/yeti/blob/master/core/observables/url.py#L20 ;

the best way is to use https://validators.readthedocs.io/en/latest/

I take a look for others data. For benkowcc; There are two types of data.

for Cruzit, just a test to check if ip is empty or not