intelmq-manager
intelmq-manager copied to clipboard
Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"}
Hello,
I've installed a fresh version of IntelMQ with PyPi but I've an issue with the connection between Intelmq-manager and intelmq-api (i guess).
I can login in the intelmq-manager but when I click on "Configuration", "Management" and "Monitor", I get this :
error : Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"}"
On "Check :
error getting check command output
On "Version":
error getting version
Error getting debugging information. Do you have IntelMQ >= 2.2.0?
i'm pretty sure it's a permission error but I don't see where I failed.
My intelmqsetup result :
Not overwriting existing 'runtime.yaml' with example.
Not overwriting existing 'harmonization.conf' with example.
Setting intelmq as owner for it's directories.
Calling `intelmqctl upgrade-config` to update/create state file.
Found previous version 3.0.1 in state file.
Checks if all harmonization fields and types are correct: Nothing to do.
Configuration upgrade successful!
Running setup for intelmq-api.
Detected Apache username 'www-data'.
Running webserver setup for intelmq-api.
Detected Apache configuration directory /etc/apache2/conf-available.
Unable to install webserver configuration api-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/api-apache.conf nor /etc/apache2/conf-available/api-apache.conf exists.
Setup of intelmq-api successful.
Running webserver setup for intelmq-manager.
Unable to install webserver configuration manager-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/manager-apache.conf nor /etc/apache2/conf-available/manager-apache.conf exists.
'intelmqsetup' completed.
I analyzed that :
Unable to install webserver configuration api-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/api-apache.conf nor /etc/apache2/conf-available/api-apache.conf exists.
275407 0 lrwxrwxrwx 1 root root 29 Sep 2 2021 intelmq-api.conf -> ../../intelmq/api-apache.conf
275419 0 lrwxrwxrwx 1 root root 33 Sep 2 2021 intelmq-manager.conf -> ../../intelmq/manager-apache.conf
root@Intelmq:~# ll /etc/intelmq/
total 60
274108 4 drwxrwxr-x 3 intelmq intelmq 4096 Aug 26 22:51 .
129793 4 drwxr-xr-x 74 root root 4096 Aug 26 23:01 ..
275402 4 -rw-rw-r-- 1 www-data www-data 506 Sep 2 2021 api-apache.conf
275403 4 -rw-rw-r-- 1 www-data www-data 241 Sep 2 2021 api-config.json
275404 4 -rw-rw-r-- 1 www-data www-data 264 Sep 2 2021 api-sudoers.conf
274109 24 -rw-rw-r-- 1 www-data www-data 21097 Sep 10 2021 harmonization.conf
276139 4 drwxrwxr-x 2 www-data intelmq 4096 Aug 26 22:51 manager
275418 4 -rw-rw-r-- 1 www-data www-data 385 Sep 2 2021 manager-apache.conf
274110 8 -rw-rw-r-- 1 www-data www-data 6313 Sep 10 2021 runtime.yaml
I've manually changed the owner (from "root" to "www-data" and "intelmq" but it doesn't work.
Thank in advance :)
There was (or is) a similar report like this, but I currently can't find it.
Hello @sebix, I saw 1 or 2 other topic with this error but the solution isn't relevant to my case unfortnately.
I removed the filter "is:issue is:open " to find solved ticket ;)
I'm fairly sure it was identical to yours. After (and despite) a successful login, the the backend did not accept the token. Which is very strange of course.
Yeah, and I'm pretty sure that it's a permission error because when it failed the first time, I tried to change, to some folder, the ownership passing through root, intelmq and www-data but with no success
If we check the ownerships: Please show a ls -la
of the directory containing the session database (path is configured in the api-config.json)
I don't know which one you need, so I "ls" both :
user@Intelmq:~$ cat /etc/intelmq/api-config.json
{
"intelmq_ctl_cmd": ["sudo", "-u", "intelmq", "intelmqctl"],
"allowed_path": "/opt/intelmq/var/lib/bots/",
"session_store": "/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi",
"session_duration": 86400,
"allow_origins": ["*"]
}
user@Intelmq:~$ ls -la /opt/intelmq/var/lib/bots/
total 12
drwxr-xr-x 3 intelmq root 4096 Aug 26 22:45 .
drwxr-xr-x 3 intelmq root 4096 Aug 26 22:45 ..
drwxr-xr-x 2 intelmq root 4096 Aug 26 23:27 file-output
user@Intelmq:~$ ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi
-rw-r--r-- 1 intelmq intelmq 24576 Aug 26 23:18 /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi
Please also ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/
(or instead of -a
-d
, to show the directory's permissions)
The last try was a mix between "root" and "intelmq" owner permission
user@Intelmq:~$ ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/
total 32
drwxr-xr-x 2 intelmq intelmq 4096 Aug 26 23:18 .
drwxr-xr-x 3 intelmq intelmq 4096 Aug 26 23:17 ..
-rw-r--r-- 1 intelmq intelmq 24576 Aug 26 23:18 intelmqapi
Try chmod g+w /var/lib/dbconfig-common/sqlite3/intelmq-api/ /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi
. The webserver (should be member of the intelmq
group) must write the database. Currently it can't.
We move forward :D
I've a new error message but we are on the good way :D
22:07:17 50× Error loading bot queues information:
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
sudo -u www-data sudo -u intelmq intelmqctl --type json list queues-and-status
22:06:30 Saved positions are not valid or not complete. The configuration has possibly been modified outside of the IntelMQ-Manager.
22:06:30 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/positions
.
22:06:29 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/bots
.
Problem is : I don't have a password for www-data, so it's maybe not the good user to use ?
Regarding my investigation, it's seems to be link to an askpass program and so, either a wrong user used or I need to add something in the sudoer file config but I'm not sure.
Your sudoers configuration for intelmq does not work properly. Is there a file in /etc/sudoers.d/
for intelmq? If not, move the api-sudoers.conf
file there and set permissions to 440.
Hello, I've a file named 01_intelmq-api :
root@Intelmq:/home/user# ls -lisa /etc/sudoers.d/
total 16
13623 4 drwxr-xr-x 2 root root 4096 Aug 26 22:51 .
129793 4 drwxr-xr-x 74 root root 4096 Sep 1 20:08 ..
19237 4 -r--r----- 1 root root 264 Sep 2 2021 01_intelmq-api
13624 4 -r--r----- 1 root root 958 Feb 27 2021 README
And the content is :
root@Intelmq:/home/user# cat /etc/sudoers.d/01_intelmq-api
# SPDX-FileCopyrightText: 2020 Birger Schacht
#
# SPDX-License-Identifier: CC0-1.0
#
# intelmq-api sudoers file, allowing the intelmq-api which usually
# is run by a webserver, to run intelmqctl as user intelmq
www-data ALL=(intelmq) NOPASSWD: /usr/bin/intelmqctl
Is the path to intelmqctl
in this file correct?
Yes it's is :)
root@Intelmq:/home/user# ls -lisa /usr/bin/intelmqctl
18325 4 -rwxr-xr-x 1 root root 964 Sep 10 2021 /usr/bin/intelmqctl
Please verify with which intelmqctl
that it is actually the file that is found when intelmqctl
is searched for in $PATH
Ohhh strange
root@Intelmq:/home/user# which intelmqctl
/usr/local/bin/intelmqctl
root@Intelmq:/home/user# ls -lisa /usr/local/bin/intelmqctl
144023 4 -rwxr-xr-x 1 root root 221 Aug 26 22:44 /usr/local/bin/intelmqctl
And the internal script isn't same :
root@Intelmq:/home/user# cat /usr/local/bin/intelmqctl
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from intelmq.bin.intelmqctl import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
root@Intelmq:/home/user# cat /usr/bin/intelmqctl
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'intelmq==3.0.2','console_scripts','intelmqctl'
import re
import sys
# for compatibility with easy_install; see #2198
__requires__ = 'intelmq==3.0.2'
try:
from importlib.metadata import distribution
except ImportError:
try:
from importlib_metadata import distribution
except ImportError:
from pkg_resources import load_entry_point
def importlib_load_entry_point(spec, group, name):
dist_name, _, _ = spec.partition('==')
matches = (
entry_point
for entry_point in distribution(dist_name).entry_points
if entry_point.group == group and entry_point.name == name
)
return next(matches).load()
globals().setdefault('load_entry_point', importlib_load_entry_point)
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')())
Running pip3 install intelmq
should only install /usr/local/bin/intelmqctl
, not /usr/bin/intelmqctl
. I don't know where you got the latter from.
Anyway, the local bindir wins, if both exist. So make sure that the path in sudoers matches the active one, and optionally get rid of one.
Yeah !! Works !!
One last error message on position xD
20:33:37 Saved positions are not valid or not complete. The configuration has possibly been modified outside of the IntelMQ-Manager.
20:33:36 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/bots
Save the configuration once, that will also set the positions in place. (if you got the file permissions right).
[Errno 13] Permission denied: '/opt/intelmq/etc/runtime.yaml'
root@Intelmq:/home/user# ls -lisa /opt/intelmq/etc/
total 52
260666 4 drwxrwxr-x 3 intelmq intelmq 4096 Sep 1 20:40 .
259699 4 drwxr-xr-x 4 intelmq root 4096 Aug 26 22:45 ..
260722 24 -rw-r--r-- 1 intelmq intelmq 21097 Aug 26 22:45 harmonization.conf
390956 4 drwxr-xr-x 2 www-data www-data 4096 Sep 1 20:40 manager
260692 8 -rw-r--r-- 1 intelmq intelmq 6530 Aug 26 22:45 runtime.yaml
259805 8 -rw-r--r-- 1 www-data www-data 6530 Sep 1 20:49 runtime.yaml.bak
I guess I need to change the owner from intelmq to www-data ?
www-data should be member of the intelmq group. And the group needs write permissions for all these files and the directories.
www-data is in the intelmq group
intelmq:x:1001:www-data
So it should have access to the runtime.yaml file arf
and in "check" panel, I've this error
error getting check command output
www-data is in the intelmq group
intelmqx1001:www-data
So it should have access to the runtime.yaml file arf
If you now gave the group write access to the file, yes.
Doesn't work ^^
When I do a intelmqctl check :
root@Intelmq:/home/user# intelmqctl check
Reading configuration files.
Checking runtime and pipeline configuration.
Checking harmonization configuration.
Checking for bots.
Traceback (most recent call last):
File "/usr/local/bin/intelmqctl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main
return x.run()
File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1048, in run
retval, results = args.func(**args_dict)
File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1564, in check
for group in utils.list_all_bots().values():
File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/utils.py", line 855, in list_all_bots
mod = importlib.import_module('.'.join(file.with_suffix('').parts))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/experts/domain_valid/expert.py", line 20, in <module>
from intelmq.lib.bot import ExpertBot
ImportError: cannot import name 'ExpertBot' from 'intelmq.lib.bot' (/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py)
root@Intelmq:/home/user# ls -lisa /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py
143598 60 -rw-r--r-- 1 root root 57581 Aug 26 22:44 /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py
Does /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py
also contain the ExpertBot
class?
May I ask why you did install with pip at all and not with the recommended deb/rpm packages?
Arf, no it doesn't.
I followed the "Installation using pip" in the intelmq-manager installation guide (https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html#installation-using-pip)
Maybe I should used the rpm package ...