Extractor fails after uploading firmware

[frakman1]

I tried to upload some firmware today and was met with this error: It is not clear what the swap limit is about. I rebooted the server to rule out any RAM issues.

[2021-11-30 17:23:07][unpack_base][ERROR]: Failed to execute docker extractor with code 1:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "./docker_extraction.py", line 68, in <module>
    sys.exit(main(_parse_args()))
  File "./docker_extraction.py", line 56, in main
    input_file = list(input_dir.iterdir())[0]
  File "/usr/lib/python3.8/pathlib.py", line 1122, in iterdir
    for name in self._accessor.listdir(self):
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/extractor/input'

Process Unpacking-Worker-3:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/FACT_core/src/helperFunctions/process.py", line 57, in run
    raise exception
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Unpacking.py", line 69, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 40, in unpack
    extracted_files = self.extract_files_from_file(file_path, tmp_dir.name)
  File "/opt/FACT_core/src/unpacker/unpack_base.py", line 31, in extract_files_from_file
    raise RuntimeError(error)
RuntimeError: Failed to execute docker extractor with code 1:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "./docker_extraction.py", line 68, in <module>
    sys.exit(main(_parse_args()))
  File "./docker_extraction.py", line 56, in main
    input_file = list(input_dir.iterdir())[0]
  File "/usr/lib/python3.8/pathlib.py", line 1122, in iterdir
    for name in self._accessor.listdir(self):
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/extractor/input'

[2021-11-30 17:23:09][process][ERROR]: Exception in Unpacking process:
Traceback (most recent call last):
  File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/scheduler/Unpacking.py", line 69, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/opt/FACT_core/src/unpacker/unpack.py", line 40, in unpack
    extracted_files = self.extract_files_from_file(file_path, tmp_dir.name)
  File "/opt/FACT_core/src/unpacker/unpack_base.py", line 31, in extract_files_from_file
    raise RuntimeError(error)
RuntimeError: Failed to execute docker extractor with code 1:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "./docker_extraction.py", line 68, in <module>
    sys.exit(main(_parse_args()))
  File "./docker_extraction.py", line 56, in main
    input_file = list(input_dir.iterdir())[0]
  File "/usr/lib/python3.8/pathlib.py", line 1122, in iterdir
    for name in self._accessor.listdir(self):
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/extractor/input'

[jstucke]

This sounds more like a path problem of FACT_docker: The extractor runs in its own docker container (to spare the user from having to install all unpackers) and the file that should be unpacked is mounted into the container under the path /tmp/extractor/input. If that file is missing, this probably means that the file path inside FACT_docker doesn't line up with the path on the host system (from where it is mounted into the FACT_docker container). I had hoped we fixed this problem already. Did you use the latest version of FACT_docker and/or rebuild the container?

[frakman1]

I did a git pull within the fact container so I thought I had the latest version.

I checked the docker inspect output and the tmp folder is mounted. I thought the extractor/input is created as part of the docker container's operation and not explicitly mounted.

       "HostConfig": {
            "Binds": [
                "/tmp/fact-docker-tmp:/tmp/fact-docker-tmp",
                "/media/data:/media/data",
                "/var/run/docker.sock:/var/run/docker.sock"
            ],

<snip>

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/media/data",
                "Destination": "/media/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/fact-docker-tmp",
                "Destination": "/tmp/fact-docker-tmp",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

The extractor directory did not exist under /tmp. I manually created it and input within but it still fails.

[jstucke]

/tmp/extractor/input is not the problem, it exists only inside the extractor container. The problem seems to be the file that is mounted there. It is mounted from inside the FACT container but it is actually on the host in the fact_fw_data folder. Mounting the file from inside the FACT container only works if the path that is mounted is the same as the "real one" outside on the host system (because we use the docker instance from the host system). Therefore, the database path inside the FACT container and the database path on the host must be the same. The start.py script should ensure this.

[frakman1]

I am not sure how to proceed. The mount paths between the host and container are specified as in the docker inspect output above. Is there something I am missing?

[jstucke]

You are right, it should be configured correctly. I will try to rebuild the container to reproduce the problem.

[jstucke]

OK I found the problem: the start.py script should mount the config directory in the FACT_docker folder (the repository folder on the host) into the container and you should make sure that temp_dir_path in FACT_docker/config/main.cfg is set to /tmp/fact-docker-tmp. The problem is, that during unpacking, the file is copied into the configured tmp folder and if it is set to /tmp, then it can't be found on the host and consequently can't be mounted into the extractor container. I'm not sure why this problem occurs now. Might be that the config folder is now mounted by default and that this wasn't the case before.

[frakman1]

Ok this is strange. I changed the temporary path in main.cfg from /tmp to /tmp/fact-docker-tmp and that seems to help, but I still get binwalk errors that look like it may also be related to the temp directory issue.

How come the latest Github project's main.cfg is still using/tmp

# Temporary Directory Path
temp_dir_path = /tmp

Do these folders need to change too?

logFile=/tmp/fact_main.log
mongoDbLogFile=/tmp/fact_mongo.log

Binwalk Error:

 Processed analysis: ['unpacker', 'file_type', 'binwalk', 'users_and_passwords', 'file_hashes', 'cpu_architecture', 'device_tree'] 
Files included: set()
Process ExceptionSafeProcess-4:872:
Traceback (most recent call last):
 File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
   self.run()
 File "/opt/FACT_core/src/helperFunctions/process.py", line 57, in run
   raise exception
 File "/opt/FACT_core/src/helperFunctions/process.py", line 52, in run
   Process.run(self)
 File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
   self._target(*self._args, **self._kwargs)
 File "/opt/FACT_core/src/analysis/PluginBase.py", line 123, in process_next_object
   finished_task = self.analyze_file(task)
 File "/opt/FACT_core/src/analysis/PluginBase.py", line 69, in analyze_file
   fo = self.process_object(file_object)
 File "/opt/FACT_core/src/plugins/analysis/binwalk/code/binwalk.py", line 33, in process_object
   result['entropy_analysis_graph'] = pic_path.read_bytes()
 File "/usr/lib/python3.8/pathlib.py", line 1229, in read_bytes
   with self.open(mode='rb') as f:
 File "/usr/lib/python3.8/pathlib.py", line 1222, in open
   return io.open(self, mode, buffering, encoding, errors, newline,
 File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
   return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/fact-docker-tmp/fact_analysis_binwalk_hm9n78fx/ea5e1237e0b88975fe76c1fd01a5eba0163aff180463c65b13cb2922ccdbf369_13900
.png'
[2021-12-02 19:03:30][PluginBase][ERROR]: Worker 1: Exception during analysis binwalk on ea5e1237e0b88975fe76c1fd01a5eba0163aff180463c65b13cb2922ccdbf369_13900 
[2021-12-02 19:03:31][Analysis][INFO]: Analysis Completed: 
UID: ea5e1237e0b88975fe76c1fd01a5eba0163aff180463c65b13cb2922ccdbf369_13900

[jstucke]

PR #696 is currently open and should hopefully fix all remaining FACT docker path issues. Also what I wrote was not entirely correct: The path in the config doesn't have to be /tmp/fact-docker-tmp but it should match the one that is passed to start.py as --docker-dir (which is /tmp/fact-docker-tmp by default if no path is provided). If you don't pass an external config directory, the tmp dir in the config inside the container should normally get patched to /tmp/fact-docker-tmp. But it could be a bad idea to use /tmp here nonetheless, because then conflicts between the temp files of your host system and the system running inside the container could occur.

[jstucke]

Since there is no activity I will close this issue. Please feel free to reopen the issue if the problem persists or if you have additional questions.