500 - Internal Server Error

[frakman1]

The web server was running fine for days, then I added a zip of some source code (curl) to test the source code static analysis plugin. I selected all plugin checkmarks. When I came back a few hours later, the webpage show the error:

Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

The docker logs showed many errors like this:

[2021-10-30 16:16:19][docker][WARNING]: [source_code_analysis]: encountered process error while processing
[2021-10-30 16:16:20][docker][WARNING]: [source_code_analysis]: encountered process error while processing
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
[2021-10-30 16:16:21][docker][WARNING]: [source_code_analysis]: encountered process error while processing

for a long time leading up to a more fatal error like this:

[2021-10-30 16:16:22][Analysis][INFO]: Analysis Completed:
UID: 914684dae82a1ef21e8343a7a5edd90850916439c1a250b9d164bd16258ec37e_3250
 Processed analysis: ['unpacker', 'file_type', 'crypto_hints', 'binwalk', 'users_and_passwords', 'printable_strings', 'malware_scanner', 'software_components', 'ip_and_uri_finder', 'crypto_material', 'qemu_exec', 'file_system_metadata', 'input_vectors', 'interesting_uris', 'string_evaluator', 'source_code_analysis', 'exploit_mitigations', 'cpu_architecture', 'elf_analysis', 'file_hashes', 'cve_lookup', 'init_systems', 'kernel_config', 'hardware_analysis', 'cwe_checker', 'known_vulnerabilities', 'tlsh']
 Files included: set()
[2021-10-30 16:16:41][db_interface_backend][ERROR]: Update of analysis failed badly (connection closed)
Process ExceptionSafeProcess-54:
[2021-10-30 16:16:41][db_interface_statistic][ERROR]: Could not store statistic backend (localhost:27018: [Errno 104] Connection reset by peer)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 291, in _receive_data_on_socket
    chunk_length = sock_info.sock.recv_into(mv[bytes_read:])
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 37, in update_statistic
    self.statistic.insert_one(content_dict)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 705, in insert_one
    self._insert(document,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 620, in _insert
    return self._insert_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 609, in _insert_one
    self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 597, in _insert_command
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 929, in _raise_connection_failure
    _raise_connection_failure(self.address, error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 247, in _raise_connection_failure
    raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: localhost:27018: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/FACT_core/src/helperFunctions/process.py", line 57, in run
    raise exception
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Analysis.py", line 391, in result_collector
    self.post_analysis(fw)
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 154, in add_analysis
    self._update_analysis(file_object, analysis_system, processed_analysis[analysis_system])
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 170, in _update_analysis
    raise exception
  File "/opt/FACT_core/src/storage/db_interface_backend.py", line 162, in _update_analysis
    collection.update_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 1028, in update_one
    self._update_retryable(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 877, in _update_retryable
    return self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 869, in _update
    return self._update(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 838, in _update
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 297, in _receive_data_on_socket
    raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed
[2021-10-30 16:16:42][process][ERROR]: Exception in Unpacking process:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 710, in command
    return command(self, dbname, spec, secondary_ok,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 152, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 200, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.8/dist-packages/pymongo/network.py", line 291, in _receive_data_on_socket
    chunk_length = sock_info.sock.recv_into(mv[bytes_read:])
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Unpacking.py", line 69, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 42, in unpack
    extracted_file_objects = self.generate_and_store_file_objects(extracted_files, Path(tmp_dir.name) / 'files', current_fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 83, in generate_and_store_file_objects
    self.db_interface.set_unpacking_lock(current_file.uid)
  File "/opt/FACT_core/src/storage/db_interface_common.py", line 314, in set_unpacking_lock
    self.locks.insert_one({'uid': uid})
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 705, in insert_one
    self._insert(document,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 620, in _insert
    return self._insert_one(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 609, in _insert_one
    self.__database.client._retryable_write(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1552, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/collection.py", line 597, in _insert_command
    result = sock_info.command(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 726, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 929, in _raise_connection_failure
    _raise_connection_failure(self.address, error)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/pool.py", line 247, in _raise_connection_failure
    raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: localhost:27018: [Errno 104] Connection reset by peer

[2021-10-30 16:16:42][db_interface_statistic][ERROR]: Could not store statistic database (connection closed)
Traceback (most recent call last):

The error seems to be related to mongodb. It looks like it stopped working at some point because I don't see port 27018 in the netstat output anymore.

I also see messages like this in the logs

[2021-10-31 00:47:06][docker][WARNING]: [source_code_analysis]: encountered process error while processing
[2021-10-31 00:47:06][docker][WARNING]: [source_code_analysis]: encountered process error while processing