Internal server error on non-ASCII characters in runtime.yaml

[creideiki]

I have installed IntelMQ and the Manager on a CentOS 8 machine, from the packages at https://download.opensuse.org/repositories/home:/sebix:/intelmq/CentOS_8/ . To that, I have added the runtime.yaml file from my working botnet.

When I navigate to /intelmq-manager/configs.html to view the botnet in Firefox, I get a 500 Internal Server Error (with no further information) when JavaScript tries to load the configuration from /intelmq/v1/api/runtime.

The Apache logs contain:

mod_wsgi (pid=737): Exception occurred processing WSGI script '/usr/lib/python3.8/site-packages/intelmq_api/intelmq-api.wsgi'., referer: http://centos8/intelmq-manager/configs.html
Traceback (most recent call last):, referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib/python3.8/site-packages/intelmq_api/intelmq-api.wsgi", line 12, in application, referer: http://centos8/intelmq-manager/configs.html
    return __hug_wsgi__(environ, start_response), referer: http://centos8/intelmq-manager/configs.html
  File "falcon/api.py", line 274, in falcon.api.API.__call__, referer: http://centos8/intelmq-manager/configs.html
  File "falcon/api.py", line 269, in falcon.api.API.__call__, referer: http://centos8/intelmq-manager/configs.html
  File "hug/api.py", line 355, in hug.api.HTTPInterfaceAPI.version_router, referer: http://centos8/intelmq-manager/configs.html
  File "hug/interface.py", line 947, in hug.interface.HTTP.__call__, referer: http://centos8/intelmq-manager/configs.html
  File "hug/interface.py", line 918, in hug.interface.HTTP.__call__, referer: http://centos8/intelmq-manager/configs.html
  File "hug/interface.py", line 840, in hug.interface.HTTP.call_function, referer: http://centos8/intelmq-manager/configs.html
  File "hug/interface.py", line 129, in hug.interface.Interfaces.__call__, referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib/python3.8/site-packages/intelmq_api/api.py", line 216, in get_runtime, referer: http://centos8/intelmq-manager/configs.html
    return utils.get_runtime(), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib/python3.8/site-packages/intelmq/lib/utils.py", line 879, in get_runtime, referer: http://centos8/intelmq-manager/configs.html
    return load_configuration(RUNTIME_CONF_FILE), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib/python3.8/site-packages/intelmq/lib/utils.py", line 216, in load_configuration, referer: http://centos8/intelmq-manager/configs.html
    config = yaml.load(fpconfig), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/site-packages/ruamel/yaml/main.py", line 329, in load, referer: http://centos8/intelmq-manager/configs.html
    constructor, parser = self.get_constructor_parser(stream), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/site-packages/ruamel/yaml/main.py", line 385, in get_constructor_parser, referer: http://centos8/intelmq-manager/configs.html
    self.reader.stream = stream, referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/site-packages/ruamel/yaml/reader.py", line 130, in stream, referer: http://centos8/intelmq-manager/configs.html
    self.determine_encoding(), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/site-packages/ruamel/yaml/reader.py", line 190, in determine_encoding, referer: http://centos8/intelmq-manager/configs.html
    self.update_raw(), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/site-packages/ruamel/yaml/reader.py", line 297, in update_raw, referer: http://centos8/intelmq-manager/configs.html
    data = self.stream.read(size), referer: http://centos8/intelmq-manager/configs.html
  File "/usr/lib64/python3.8/encodings/ascii.py", line 26, in decode, referer: http://centos8/intelmq-manager/configs.html
    return codecs.ascii_decode(input, self.errors)[0], referer: http://centos8/intelmq-manager/configs.html
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128), referer: http://centos8/intelmq-manager/configs.html

That byte position contains an "å", UTF-8 coded (U+00e5, 0xc3 0xa5), which is not encodable in US-ASCII.

None of the command-line tools (e.g. intelmqctl) have ever had a problem with this.

Copying the minimal code from the libraries and running it in a REPL does not replicate the error, even when started with LC_CTYPE=C:

[root@centos8 intelmq]# env LC_CTYPE=C python3.8
Python 3.8.8 (default, Aug 25 2021, 16:13:02) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-3)] on linux

from ruamel.yaml import YAML
yaml = YAML(typ="unsafe", pure=True)
configuration_filepath = 'runtime.yaml'
with open(configuration_filepath, 'r') as fpconfig:
   config = yaml.load(fpconfig)
print(len(config))

19

So this looks like some sort of problem with the environment that the API is running in.

A quick-and-dirty fix to make it work right now was to change the code that opens the file, in intelmq/lib/utils.py:load_configuration (https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/lib/utils.py#L216), to open the file as binary (mode 'rb' instead of 'r'), making the python objects read from it bytes instead of string. However, I have no idea how that would affect a system where that actually makes a difference, e.g. Windows.

[sebix]

A better solution would be to open the file as UTF-8 with codecs.open(filename, encoding='UTF-8'), and the API backend must set the encoding accordingly.