v2.1.2 default installation after instruction lacks dependencies for parsers/html_table

[bernhardreiter]

Following v2.1.2 Install and Development instructions, then starting the example default bot net, the feodo-tracker-browse-parser stopped with

  File "/opt/dev_intelmq/intelmq/bots/parsers/html_table/parser.py", line 37, in init
    raise MissingDependencyError("beautifulsoup4")
intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'beautifulsoup4', please install it with apt/yum/dnf/zypper (possibly named python3-beautifulsoup4) or pip3.

Expectation: the instructions or the setup process should make sure that the dependencies for the standard bots are there.

Workaround on Ubuntu 18.04 LTS:

apt-install python-bs4 python3-lxml

Idea: the intelmqctl check command should also check if dependencies can be loaded. Probably needs changes in the bots implementation.

[ghost]

Sorry for the accidental close, I mixed up the issue numbers -.-

[ghost]

See also #552

Requires some changes in the core. The requirements of bots need to be defined somewhere

[motok]

intelmq 3.0.1 seems still have this issue.

$ intelmqctl -v
3.0.1

2022-01-18 16:30:35,475 - feodo-tracker-browse-parser - INFO - HTMLTableParserBot initialized with id feodo-tracker-browse-parser and intelmq 3.0.1 and python 3.8.12 (default, Nov  4 2021, 01:12:24) as process 769.
2022-01-18 16:30:35,475 - feodo-tracker-browse-parser - INFO - Bot is starting.
2022-01-18 16:30:35,650 - feodo-tracker-browse-parser - ERROR - Bot initialization failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/intelmq/lib/bot.py", line 209, in __init__
    self.init()
  File "/usr/local/lib/python3.8/site-packages/intelmq/bots/parsers/html_table/parser.py", line 55, in init
    raise MissingDependencyError("beautifulsoup4")
intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'beautifulsoup4', please install it with apt/yum/dnf/zypper (possibly named python3-beautifulsoup4) or pip3.
2022-01-18 16:30:35,653 - feodo-tracker-browse-parser - INFO - Bot stopped.

[sebix]

Yes, it's the case for all bots which have specific additional requirements.

[waldbauer-certat]

This doesnt seems like a bug to me, because the error message should be clear enough. Maybe add some better error handling in intelmqctl and intelmq-manager if the error is MissingDependencyError.

Would remove the bug label, as its not a bug. @sebix

[sebix]

It's definitely bad usability. IIRC the reason I labelled this as bug was that this particular bot is part of the standard botnet. In other words: The default installation does not work without installing further dependencies. If you disagree, feel free to remove the label.

https://github.com/certtools/intelmq/blob/469dc4951a0d39c1b3f62bee156c2b371f7d1c0c/intelmq/etc/runtime.yaml#L60-L65

[kamil-certat]

I suggest disabling those bots in the default config. The bs4 is a heavy dependency, and I don't think it makes sense to ship it for everyone. We may find other alternative bots to present the duplicator usage, but so far, shipping not working config by default doesn't make any sense.