Some plugins can't run because they can't be "installed" first.

[gluesmith2021]

Observed Issue

After following instruction from readme, FACT does run but some analysis plugins don't because they require docker images that can't be pulled. For instance, file_system_metadata fails with the following error:

[2023-09-25 13:26:16][docker][WARNING]: [file_system_metadata]: encountered docker error while processing
Process ExceptionSafeProcess-22:8:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 819, in run
    container = self.create(image=image, command=command,
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 878, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 428, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 439, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/containers/create: Not Found ("No such image: fact/fs_metadata:latest")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/create?tag=latest&fromImage=fact%2Ffs_metadata
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/FACT_core/src/helperFunctions/process.py", line 56, in run
    raise exception
  File "/opt/FACT_core/src/helperFunctions/process.py", line 51, in run
    Process.run(self)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/FACT_core/src/analysis/PluginBase.py", line 140, in process_next_object
    finished_task = self.analyze_file(task)
  File "/opt/FACT_core/src/analysis/PluginBase.py", line 87, in analyze_file
    fo = self.process_object(file_object)
  File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 65, in process_object
    self._extract_metadata(file_object)
  File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 93, in _extract_metadata
    self._extract_metadata_from_file_system(file_object)
  File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 104, in _extract_metadata_from_file_system
    output = self._mount_in_docker(tmp_dir)
  File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 114, in _mount_in_docker
    result = run_docker_container(
  File "/opt/FACT_core/src/helperFunctions/docker.py", line 35, in run_docker_container
    container = client.containers.run(image, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 822, in run
    self.client.images.pull(image, platform=platform)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/images.py", line 444, in pull
    pull_log = self.client.api.pull(
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 428, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/create?tag=latest&fromImage=fact%2Ffs_metadata: Not Found ("pull access denied for fact/fs_metadata, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
[2023-09-25 13:26:16][PluginBase][ERROR]: Worker 0: Exception during analysis file_system_metadata on 0bd402cc5ef97c260e7b1486da0f841b84ffe8c0867dd48e08083b8f6d2e1806_21528576

Identified Cause

Plugins never get a chance to be installed by start.py pull. So for the above plugin example, the required local fact/fs_metadata docker image is never built by the plugin code as it should.

This happens because distribution_check(allow_unsupported=False) is called for any plugin installer being instantiated. distribution_check then calls sys.exit(1) because fact-core-script docker image is based on the "unsupported" Alpine.

In details

(code snippets below are from fact-core-script image):

• start.py pull on the host runs /opt/FACT_core/src/install.py --backend-docker-images --frontend-docker-images in fact-core-script container.
• In install.py, method install() has the following :

  only_docker = not skip_docker and none_chosen and (args.backend_docker_images or args.frontend_docker_images)
  
  # When just pulling the docker images we don't depend on anything distribution specific
  distribution = check_distribution(allow_unsupported=only_docker)
  

• Here, skip_docker=False, none_chosen=True, and both checked args.xyz are True as well.
  • Therefore, check_distribution emits warnings about the unsupported distribution (because the script docker image uses an "unsupported" Alpine), that we can ignore (as mentioned in issue #24 among others)
• Further down it calls install_docker_images() which in turns call
  • backend_install_docker_images(): runs fine
  • backend_install_plugin_docker_images(): fails!
• The latter call, at some point: _install_plugins() in install/backend.py with

  _install_plugins(distribution=None, skip_docker=False, only_docker=True)
  

• This finally calls install_docker_images() for each plugin found in the plugins folder.

Now, in the above example, with file_system_metadata plugin, there is:

class FileSystemMetadataInstaller(AbstractPluginInstaller):
    base_path = Path(__file__).resolve().parent

    def install_docker_images(self):
        self._build_docker_image('fact/fs_metadata:latest')

The base class AbstractPluginInstaller is initialized with:

    def __init__(self, distribution: Optional[str] = None, skip_docker: bool = skip_docker_env):
        self.distribution = distribution or check_distribution()
        self.build_path = self.base_path / 'build'
        self.skip_docker = skip_docker

Here distribution=None so check_distribution() is called with default parameters, that is check_distribution(allow_unsupported=False)

Then the distribution check fails with

        logging.critical(msg)
        sys.exit(1)

This obviously occurs on the first plugin installer that is instantiated, and the whole installation aborts at that point.

While a human could ignore "unsupported distribution" warnings as mentioned in #24 , the installer does not, and aborts. It does not seem to be the "intended behavior" as some plugins can't run if they are not previously installed from the above.