IntelOwl
IntelOwl copied to clipboard
Published
20 hours ago •
intelowlproject
intelowlproject
fixes #1699 Permhash analyzer
(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example: <feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)
Description
Closes #1699 New analyzer: Permhash
Type of change
New analyzer
Please delete options that are not relevant.
- [ ] Bug fix (non-breaking change which fixes an issue).
- [x] New feature (non-breaking change which adds functionality).
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).
Checklist
- [x] I have read and understood the rules about how to Contribute to this project
- [x] The pull request is for the branch
develop - [x] A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
- [x] I strictly followed the documentation "How to create a Plugin"
- [x] Usage file was updated.
- [ ] Advanced-Usage was updated (in case the plugin provides additional optional configuration).
- [x] If the plugin requires mocked testing,
_monkeypatch()was used in its class to apply the necessary decorators. - [x] I have dumped the configuration from Django Admin using the
dumpplugincommand and added it in the project as a data migration. ("How to share a plugin with the community") - [x] If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive
test_files.zipand you added the default tests for that mimetype in test_classes.py. - [x] If you created a new analyzer and it is free (does not require API keys), please add it in the
FREE_TO_USE_ANALYZERSplaybook by following this guide. - [x] Check if it could make sense to add that analyzer/connector to other freely available playbooks.
- [x] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
- [ ] I have added that raw JSON sample to the
MockUpResponseof the_monkeypatch()method. This serves us to provide a valid sample for testing.
- [ ] If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
- [ ] If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
- [x] Linters (
Black,Flake,Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf. - [x] I have added tests for the feature/bug I solved (see
testsfolder). All the tests (new and old ones) gave 0 errors. - [ ] If the GUI has been modified:
- [ ] I have a provided a screenshot of the result in the PR.
- [ ] I have created new frontend tests for the new component or updated existing ones.
- [x] After you had submitted the PR, if
DeepSource,Django Doctorsor other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.
Important Rules
- If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
- Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.
Output screenshot
JSON output
{
"hash":"aad106ceb64ac2a636ddec77c3feed4c2ffc5c27ab353660d8cb3e1c971df277"
}
Hey @mlodic I have added new mimetypes for .crx and .json but migration shows error. Here are the migration error logs:
intelowl_uwsgi | Applying analyzers_manager.0066_analyzer_config_phoneinfoga... OK
intelowl_uwsgi | Applying analyzers_manager.0067_update_misp... OK
intelowl_uwsgi | Applying analyzers_manager.0068_analyzer_config_feodo_tracker... OK
intelowl_uwsgi | Applying analyzers_manager.0069_analyzer_config_bgp_ranking... OK
intelowl_uwsgi | Applying analyzers_manager.0070_urlhaus_threatfox_disable_param... OK
intelowl_uwsgi | Applying analyzers_manager.0071_analyzer_config_tor_nodes_danmeuk... OK
intelowl_uwsgi | Applying analyzers_manager.0072_analyzer_config_tweetfeed... OK
intelowl_uwsgi | Applying analyzers_manager.0073_remove_dragonfly_analyzer... OK
intelowl_uwsgi | Applying analyzers_manager.0074_adjust_maximum_tlp... OK
intelowl_uwsgi | Applying analyzers_manager.0075_adjust_greynoise... OK
intelowl_uwsgi | Applying analyzers_manager.0076_analyzer_config_greynoise_labs... OK
intelowl_uwsgi | Applying analyzers_manager.0077_analyzer_config_abusix... OK
intelowl_uwsgi | Traceback (most recent call last):
intelowl_uwsgi | File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 54, in _create_object
intelowl_uwsgi | o = Model.objects.get(**no_mtm)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
intelowl_uwsgi | return getattr(self.get_queryset(), name)(*args, **kwargs)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
intelowl_uwsgi | raise self.model.DoesNotExist(
intelowl_uwsgi | __fake__.AnalyzerConfig.DoesNotExist: AnalyzerConfig matching query does not exist.
intelowl_uwsgi |
intelowl_uwsgi | During handling of the above exception, another exception occurred:
intelowl_uwsgi |
intelowl_uwsgi | Traceback (most recent call last):
intelowl_uwsgi | File "/opt/deploy/intel_owl/manage.py", line 25, in <module>
intelowl_uwsgi | main()
intelowl_uwsgi | File "/opt/deploy/intel_owl/manage.py", line 21, in main
intelowl_uwsgi | execute_from_command_line(sys.argv)
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
intelowl_uwsgi | utility.execute()
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
intelowl_uwsgi | self.fetch_command(subcommand).run_from_argv(self.argv)
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
intelowl_uwsgi | self.execute(*args, **cmd_options)
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
intelowl_uwsgi | output = self.handle(*args, **options)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
intelowl_uwsgi | res = handle_func(*args, **kwargs)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/opt/deploy/intel_owl/api_app/management/commands/migrate.py", line 8, in handle
intelowl_uwsgi | super().handle(*args, **options)
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
intelowl_uwsgi | res = handle_func(*args, **kwargs)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
intelowl_uwsgi | post_migrate_state = executor.migrate(
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
intelowl_uwsgi | state = self._migrate_all_forwards(
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
intelowl_uwsgi | state = self.apply_migration(
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
intelowl_uwsgi | state = migration.apply(state, schema_editor)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
intelowl_uwsgi | operation.database_forwards(
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
intelowl_uwsgi | self.code(from_state.apps, schema_editor)
intelowl_uwsgi | File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 72, in migrate
intelowl_uwsgi | exists = _create_object(Model, plugin)
intelowl_uwsgi | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_uwsgi | File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0078_analyzer_config_permhash.py", line 57, in _create_object
intelowl_uwsgi | o.full_clean()
intelowl_uwsgi | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1502, in full_clean
intelowl_uwsgi | raise ValidationError(errors)
intelowl_uwsgi | django.core.exceptions.ValidationError: {'supported_filetypes': ["Item 6 in the array did not validate: Value 'application/x-chrome-extension' is not a valid choice."]}
intelowl_uwsgi | Applying analyzers_manager.0078_analyzer_config_permhash...Issue with migration exiting
intelowl_uwsgi exited with code 1
dependency failed to start: container intelowl_uwsgi exited (1)
This analyzer is not working when I tried testing it in the frontend. This is what I get (although other analyzers are working fine):
if you don't show the logs I don't know how to help. Also, please pull from develop that you have conflicts
you also need to change the test_classes.py if you add a new filetype and add a new test file for that type
2024-04-11 17:20:39,528 - api_app.models - set_final_status - INFO - <Job: Job(#1, "12.1.2.3")> setting status to reported_without_fails
2024-04-11 17:20:39,534 - api_app.websocket - serialize_and_send_job - DEBUG - send data for the job: 1 to the groups: ['job-1', 'job-1-perm']
2024-04-11 17:20:39,535 - api_app.websocket - serialize_and_send_job - DEBUG - send data to the group: job-1
2024-04-11 17:20:39,622 - api_app.websocket - serialize_and_send_job - DEBUG - send data to the group: job-1-perm
2024-04-11 17:21:54,451 - api_app.views - analyze_multiple_files - INFO - received analyze_multiple_files from user moon
2024-04-11 17:21:54,465 - api_app.serializers.job - validate - DEBUG - before attrs: OrderedDict([('user', <SimpleLazyObject: <User: moon>>), ('is_sample', True), ('tlp', 'AMBER'), ('analyzers_requested', [<AnalyzerConfig: Permhash>]), ('connectors_requested', []), ('tags_labels', <generator object _AbstractJobCreateSerializer.validate_tags_labels at 0x7fe69c27f840>), ('scan_mode', 2), ('scan_check_time', datetime.timedelta(days=1)), ('investigation', None), ('file', <InMemoryUploadedFile: bitbar-sample-app.apk (application/vnd.android.package-archive)>)])
2024-04-11 17:21:54,532 - api_app.serializers.job - validate - DEBUG - after attrs: OrderedDict([('user', <SimpleLazyObject: <User: moon>>), ('is_sample', True), ('tlp', 'AMBER'), ('analyzers_requested', [<AnalyzerConfig: Permhash>]), ('connectors_requested', []), ('scan_mode', 2), ('scan_check_time', datetime.timedelta(days=1)), ('investigation', None), ('file', <InMemoryUploadedFile: bitbar-sample-app.apk (application/vnd.android.package-archive)>), ('file_name', 'bitbar-sample-app.apk'), ('file_mimetype', MimeTypes.APK), ('md5', '00cc5435151aa38a091781922c0390a4'), ('analyzers_to_execute', [<AnalyzerConfig: Permhash>]), ('connectors_to_execute', []), ('visualizers_to_execute', []), ('warnings', []), ('tags', <generator object _AbstractJobCreateSerializer.validate_tags_labels at 0x7fe69c27f840>)])
2024-04-11 17:21:54,533 - api_app.serializers.job - check_previous_jobs - INFO - Checking previous jobs
2024-04-11 17:21:54,604 - api_app.serializers.job - create - INFO - Job 2 created
2024-04-11 17:21:54,604 - api_app.serializers.job - create - INFO - Sending task for job 2
2024-04-11 17:21:54,626 - api_app.views - analyze_multiple_files - INFO - finished analyze_multiple_files from user moon
2024-04-11 17:21:55,943 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:21:56,387 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:22:17,056 - api_app.mixins - list - DEBUG - View list_ConnectorConfig_moon_1_70 cache hit
2024-04-11 17:22:17,085 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_1_70 cache hit
2024-04-11 17:22:17,185 - api_app.mixins - list - DEBUG - View list_PivotConfig_moon_1_70 cache hit
2024-04-11 17:22:17,317 - api_app.mixins - list - DEBUG - View list_VisualizerConfig_moon_1_70 cache hit
2024-04-11 17:22:17,419 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
2024-04-11 17:22:17,478 - api_app.mixins - list - DEBUG - View list_IngestorConfig_moon_1_70 cache hit
2024-04-11 17:22:17,584 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_2_70 cache hit
2024-04-11 17:22:17,760 - api_app.mixins - list - DEBUG - View list_AnalyzerConfig_moon_3_70 cache hit
2024-04-11 17:22:17,864 - api_app.mixins - list - DEBUG - View list_PlaybookConfig_moon_1_70 cache hit
2024-04-11 17:22:21,377 - api_app.views - get_queryset - INFO - user: moon request the jobs with params: <QueryDict: {}>
@mlodic
Actually, I solved it myself. There was some issue with the latest version of permhash library (v0.1.4.2). I used v0.1.4 and it worked perfectly fine.