Multithreading causes bot to crash or not start

[DigitalTrustCenter]

We are testing the mulithreading capabilities of IntelMQ in combination with RabbitMQ. We have found two (regression) issues that completely break mulithreaded bots:

1. The new YAML configuration format uses a global YAML object in lib/utils.py to read/write configuration. When mulithreading is enabled for a bot, each thread calls _init_ on the bot class. Since the init function loads the configuration of the bot, this means that each thread tries to load the configuration at the same time. This causes a race condition on the YAML loader, and usually crashes the threads, and thus the bot fails to start.

2. When starting new threads for a bot, the parameter start=True is passed to the _init_ method of the Bot class. This causes the bot at the end of the _init_ method to call the start function, and thus start processing messages. The problem lies in the fact that the subclasses for Bot (CollectorBot, ParserBot, ExpertBot, OutputBot) do not pass the start parameter to the (super) Bot class. This defaults the start parameter to False, hence the threads never start, and quit without doing anything.

[sebix]

Merging is blocked by #2237

[DigitalTrustCenter]

Any update on this?

We just had the same problem with the YAML loader in the API and webinput CSV. We are using Apache + mod WSGI for both the API and webinput csv.It looks like mod WSGI shares the utils globals (like the YAML loader) among those two applications. One user was opening the IntelMQ web gui (loading the runtime config via the API), while the other was submitting a CSV (loading the harmonization config). Both calls failed, due to the YAML loader being shared, and trying to load two files at the same time.

[sebix]

ping @waldbauer-certat for review of https://github.com/certtools/intelmq/pull/2237

[sebix]

rebased on master

[codecov-commenter]

Codecov Report

Merging #2236 (ec94bf2) into develop (427fe01) will decrease coverage by 0.47%. The diff coverage is 83.33%.

:exclamation: Current head ec94bf2 differs from pull request most recent head 79ac816. Consider uploading reports for the commit 79ac816 to get more accurate results

@@             Coverage Diff             @@
##           develop    #2236      +/-   ##
===========================================
- Coverage    76.89%   76.41%   -0.48%     
===========================================
  Files          454      453       -1     
  Lines        24108    23963     -145     
  Branches      3515     3781     +266     
===========================================
- Hits         18537    18312     -225     
- Misses        4798     4908     +110     
+ Partials       773      743      -30     

Impacted Files	Coverage Δ
intelmq/lib/utils.py	72.29% <50.00%> (-0.85%)	:arrow_down:
intelmq/lib/bot.py	64.05% <100.00%> (+0.26%)	:arrow_up:
intelmq/bin/intelmqdump.py	18.30% <0.00%> (-24.56%)	:arrow_down:
intelmq/bin/intelmqctl.py	9.18% <0.00%> (-4.34%)	:arrow_down:
intelmq/lib/processmanager.py	23.99% <0.00%> (-3.82%)	:arrow_down:
intelmq/bots/collectors/opendxl/collector.py	30.15% <0.00%> (-3.18%)	:arrow_down:
intelmq/bots/experts/mcafee/expert_mar.py	42.22% <0.00%> (-2.23%)	:arrow_down:
intelmq/bots/collectors/tcp/collector.py	84.31% <0.00%> (-1.97%)	:arrow_down:
intelmq/bots/experts/reverse_dns/expert.py	82.25% <0.00%> (-1.87%)	:arrow_down:
... and 40 more