ail-framework
ail-framework copied to clipboard
Published
20 hours ago •
ail-project
ail-project
AIL Framework not starting v4.2 [Ubuntu 20.04]
Hello AIL Framework team, I have a problem running AIL Framework, the Website doesn't start.
I installed pybgpranking manually because it failed and I updated jsonschema to 4.9.1
ERROR: pymisp 2.4.160.1 has requirement jsonschema<5.0.0,>=4.9.1, but you'll have jsonschema 3.2.0 which is incompatible.
After check the screens I detected:
- Flask_AIL ERROR:
Flask_base_template.py create_default_user.py server.key
Flask_server.py create_new_web_module.py static
__pycache__ modules templates
blueprints server.crt update_thirdparty.sh
Traceback (most recent call last):
File "./Flask_server.py", line 24, in <module>
from pytaxonomies import Taxonomies
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
from .api import Taxonomies, Taxonomy, Predicate, Entry
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
import requests
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
from . import packages, utils
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/utils.py'
- Script_AIL ERROR:
Flask_base_template.py create_default_user.py server.key
Flask_server.py create_new_web_module.py static
__pycache__ modules templates
blueprints server.crt update_thirdparty.sh
Traceback (most recent call last):
File "./Flask_server.py", line 24, in <module>
from pytaxonomies import Taxonomies
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
from .api import Taxonomies, Taxonomy, Predicate, Entry
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
import requests
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
from . import packages, utils
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/sugar/poll.py'
- Redis_AIL info:
52383:M 10 Aug 2022 22:27:17.193 # Server initialized
52383:M 10 Aug 2022 22:27:17.193 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
52383:M 10 Aug 2022 22:27:17.193 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
52383:M 10 Aug 2022 22:27:17.193 * Ready to accept connections
- Other screen without error messages !
PS: AIL Starting with ./LAUNCH.sh -l output without ERRORS:
* Launching Redis servers
* Launching ARDB servers
* Launching logging process
* Launching all the queues
* Checking configuration
Config File: Nothing to update
Config File: Nothing to update
* Configuration up-to-date
* Launching core scripts ...
* Launching scripts
* Launching Flask server
Thank you
BTW the errors change every time when I run LAUNCH -k and LAUNCH -l
- SOME STOP + START ->
Flask_base_template.py create_new_web_module.py static
Flask_server.py modules templates
blueprints server.crt update_thirdparty.sh
create_default_user.py server.key
Traceback (most recent call last):
File "./Flask_server.py", line 16, in <module>
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/__init__.py", line 4, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/json/__init__.py'
- Another STOP + START ->
Flask_base_template.py create_new_web_module.py static
Flask_server.py modules templates
blueprints server.crt update_thirdparty.sh
create_default_user.py server.key
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/usr/lib/python3.8/site.py", line 597, in <module>
main()
File "/usr/lib/python3.8/site.py", line 580, in main
known_paths = venv(known_paths)
File "/usr/lib/python3.8/site.py", line 498, in venv
with open(virtual_conf, encoding='utf-8') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/pyvenv.cfg'
And to finish ./LAUNCH -t command output:
ubuntu@ailframework:~/ailframework/bin$ ./LAUNCH.sh -t
EEE
======================================================================
ERROR: Failure: PermissionError ([Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/testApi.py", line 17, in <module>
from Flask_server import app
File "/home/ubuntu/ailframework/var/www/Flask_server.py", line 160, in <module>
importlib.import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/var/www/modules/Tags/Flask_Tags.py", line 39, in <module>
clusters = Clusters(skip_duplicates=True)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/api.py", line 294, in __init__
with open(cluster_file, 'r') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json'
-------------------- >> begin captured logging << --------------------
pymisp: DEBUG: pymisp loaded properly
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/testHelper.py", line 9, in <module>
from Helper import Process
File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
import zmq
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
from zmq import backend
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
raise original_error from None
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
_ns = select_backend(first)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
mod = import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory
======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/test_modules.py", line 15, in <module>
from modules.ApiKey import ApiKey
File "/home/ubuntu/ailframework/bin/modules/ApiKey.py", line 21, in <module>
from modules.abstract_module import AbstractModule
File "/home/ubuntu/ailframework/bin/modules/abstract_module.py", line 17, in <module>
from Helper import Process
File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
import zmq
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
from zmq import backend
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
raise original_error from None
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
_ns = select_backend(first)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
mod = import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory
Name Stmts Miss Cover
----------------------------------------------------------
bin/BankAccount.py 72 72 0%
bin/Crawler.py 281 281 0%
bin/Cryptocurrencies.py 78 78 0%
bin/Cve.py 26 26 0%
bin/DumpValidOnion.py 23 23 0%
bin/Duplicates.py 116 116 0%
bin/Helper.py 158 153 3%
bin/IPAddress.py 50 50 0%
bin/Languages.py 21 21 0%
bin/MISP_The_Hive_feeder.py 156 156 0%
bin/Mail.py 135 135 0%
bin/Mixer.py 114 114 0%
bin/ModuleStats.py 80 80 0%
bin/ModulesInformationV2.py 549 549 0%
bin/NotificationHelper.py 52 52 0%
bin/PgpDump.py 164 164 0%
bin/QueueIn.py 13 13 0%
bin/QueueOut.py 14 14 0%
bin/Release.py 46 46 0%
bin/SourceCode.py 39 39 0%
bin/Tools.py 62 62 0%
bin/Update-conf.py 59 59 0%
bin/Update.py 329 329 0%
bin/__init__.py 0 0 100%
bin/ailleakObject.py 90 90 0%
bin/core/ail_2_ail.py 852 697 18%
bin/core/screen.py 122 105 14%
bin/empty_queue.py 18 18 0%
bin/export/AILObjects.py 39 22 44%
bin/export/Export.py 49 26 47%
bin/export/MispExport.py 331 276 17%
bin/export/MispImport.py 172 140 19%
bin/import/importer.py 61 34 44%
bin/indexer_lookup.py 60 60 0%
bin/launch_queues.py 49 49 0%
bin/lib/ConfigLoader.py 41 11 73%
bin/lib/Correlate_object.py 371 327 12%
bin/lib/Decoded.py 260 210 19%
bin/lib/Domain.py 573 463 19%
bin/lib/Investigations.py 294 219 26%
bin/lib/Language.py 27 18 33%
bin/lib/Screenshot.py 161 120 25%
bin/lib/Tracker.py 944 799 15%
bin/lib/Username.py 13 1 92%
bin/lib/btc_ail.py 40 31 22%
bin/lib/crawlers.py 1009 833 17%
bin/lib/d4.py 58 39 33%
bin/lib/domain_basic.py 16 5 69%
bin/lib/exceptions.py 8 2 75%
bin/lib/item_basic.py 161 117 27%
bin/lib/objects/CryptoCurrencies.py 34 19 44%
bin/lib/objects/Decodeds.py 40 20 50%
bin/lib/objects/Domains.py 33 17 48%
bin/lib/objects/Items.py 426 319 25%
bin/lib/objects/Pgps.py 30 15 50%
bin/lib/objects/Screenshots.py 24 8 67%
bin/lib/objects/Usernames.py 31 16 48%
bin/lib/objects/abstract_object.py 57 31 46%
bin/lib/objects/ail_objects.py 54 27 50%
bin/lib/queues_modules.py 53 34 36%
bin/modules/ApiKey.py 50 44 12%
bin/modules/Categ.py 49 49 0%
bin/modules/Credential.py 111 111 0%
bin/modules/CreditCards.py 40 40 0%
bin/modules/Decoder.py 99 99 0%
bin/modules/DomClassifier.py 51 51 0%
bin/modules/Global.py 122 122 0%
bin/modules/Hosts.py 31 31 0%
bin/modules/Indexer.py 89 89 0%
bin/modules/Keys.py 116 116 0%
bin/modules/LibInjection.py 57 57 0%
bin/modules/Onion.py 61 61 0%
bin/modules/Phone.py 40 40 0%
bin/modules/SQLInjectionDetection.py 48 48 0%
bin/modules/SentimentAnalysis.py 98 98 0%
bin/modules/Tags.py 26 26 0%
bin/modules/Telegram.py 53 53 0%
bin/modules/Urls.py 40 40 0%
bin/modules/Zerobins.py 30 30 0%
bin/modules/__init__.py 0 0 100%
bin/modules/abstract_module.py 52 46 12%
bin/modules/submit_paste.py 189 189 0%
bin/packages/Correlation.py 266 213 20%
bin/packages/Cryptocurrency.py 68 49 28%
bin/packages/Date.py 127 101 20%
bin/packages/Hash.py 30 19 37%
bin/packages/HiddenServices.py 231 190 18%
bin/packages/Import_helper.py 56 43 23%
bin/packages/Item.py 401 298 26%
bin/packages/Paste.py 195 146 25%
bin/packages/Pgp.py 17 5 71%
bin/packages/Tag.py 483 396 18%
bin/packages/Term.py 392 325 17%
bin/packages/User.py 40 24 40%
bin/packages/__init__.py 0 0 100%
bin/packages/git_status.py 110 95 14%
bin/packages/lib_refine.py 94 94 0%
bin/packages/lib_words.py 51 51 0%
bin/template.py 18 18 0%
bin/tor_fetcher.py 46 46 0%
bin/update-background.py 44 44 0%
----------------------------------------------------------
TOTAL 13359 11447 14%
----------------------------------------------------------------------
Ran 3 tests in 0.002s
FAILED (errors=3)
Thank you
Other error details in the installating_steps.sh step:
Successfully built pymispgalaxies
Installing collected packages: jsonschema, pymispgalaxies
Attempting uninstall: jsonschema
Found existing installation: jsonschema 4.9.1
Uninstalling jsonschema-4.9.1:
Successfully uninstalled jsonschema-4.9.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymisp 2.4.160.1 requires jsonschema<5.0.0,>=4.9.1, but you have jsonschema 3.2.0 which is incompatible.
Successfully installed jsonschema-3.2.0 pymispgalaxies-0.2
And later ->
Installing collected packages: jsonschema
Attempting uninstall: jsonschema
Found existing installation: jsonschema 3.2.0
Uninstalling jsonschema-3.2.0:
Successfully uninstalled jsonschema-3.2.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymispgalaxies 0.2 requires jsonschema<4.0,>=3.2, but you have jsonschema 4.9.1 which is incompatible.
Successfully installed jsonschema-4.9.1
Requirement already satisfied: thehive4py in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (1.8.1)
Requirement already satisfied: python-magic in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (from thehive4py) (0.4.27)
It looks that some package require one jsonschema version and another package requires a different version
FYI, the same results using python 3.8 and python 3.9, both using Ubuntu 20.04
Looking at the error message, I guess it's a mix between the ZMQ library and especially the underlying binding.
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory
For example libzmq5 would be need to be installed.
Hello @adulau,
I'll check it deeply, but two days ago I removed the folder "4.2" thinking to install version "4.1" and it worked but the "update/current_version" file says "v4.2" even when folder "4.2" doesn't exist, but I don't care if it works \0/ I need to do more "tests" and let the installation process "strong" and with sense, maybe the days ago some "git dependency" was updated and it made the installation work ...
Server Management AIL Framework status info:
AIL Version | None [(release note)](https://github.com/ail-project/ail-framework/releases/tag/None)
Current Branch | master
Current Commit ID | 3b902020cc6a2c4770c83c7b3b2bca950402d5dc
Current Tag | v4.2
BTW I commented the next lines in the file installing_deps.sh: (installation fails with them)
125: shallow clone
126: git fetch --depth=500 --tags --prune
127: if [ ! -z "$TRAVIS" ]; then
128: echo "Travis detected"
129: git fetch --unshallow
130: fi
Another thing that I detected is that if I want to access to the "Server Management" menu I need to include the github.com ssh-key into ssh_known_hosts file, here the output of the screen "Flask_AIL":
10.215.231.1 - - [27/Aug/2022 16:50:34] "GET /PasteSubmit/submit_status?UUID= HTTP/1.1" 200 -
10.215.231.1 - - [27/Aug/2022 16:50:34] "GET /Tags/get_all_tags_galaxies HTTP/1.1" 200 -
10.215.231.1 - - [27/Aug/2022 16:50:34] "GET /Tags/get_all_tags_taxonomies HTTP/1.1" 200 -
The authenticity of host 'github.com (140.82.121.4)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
When you accept "yes" or your add the ssh-key of github.com to he file /etc/ssh/ssh_known_hosts then load the "Server Management" page and follow with the file "/static/css/bootstrap4.min.css" and so on:
The authenticity of host 'github.com (140.82.121.4)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 10.215.231.1 - - [27/Aug/2022 16:50:41] "GET /settings/ HTTP/1.1" 200 -
10.215.231.1 - - [27/Aug/2022 16:50:41] "GET /static/css/bootstrap4.min.css HTTP/1.1" 304 -
10.215.231.1 - - [27/Aug/2022 16:50:41] "GET /static/css/font-awesome.min.css HTTP/1.1" 304 -
10.215.231.1 - - [27/Aug/2022 16:50:41] "GET /static/js/jquery.js HTTP/1.1" 304 -
10.215.231.1 - - [27/Aug/2022 16:50:41] "GET /static/js/popper.min.js HTTP/1.1" 304 -
Thanks for your help and I will let you know if I find more inconvenients