Issues when fixing the dependencies inside Flatpak apps

[febrezo]

• face_recognition version: any
• Python version: 3.7
• Operating System: Flatpak

Description

I'm trying to build face_recognition within a Flatpak to ship it with applications that use it but I'm struggling with making it work. Everything seems to crash when installing the dlib dependency.

I was trying to emulate the way in which you proposed the Dockerfile with the dependencies fixed in a Flatpak, since doing so is really interesting to make applications using it trully portable amongst different operating systems.

What I Did

I started with a simple manifest file which simply creates a flatpak for a hello.sh file as follows:

#!/bin/sh
echo "Hello world, from inside of Flatpak sandbox"
python3 --version
python3 -c "import face_recognition"

My com.felixbrezo.FlatpakTest.json manifest file is this one:

{
    "app-id": "com.felixbrezo.FlatpakTest",
    "runtime": "org.freedesktop.Platform",
    "runtime-version": "19.08",
    "sdk": "org.freedesktop.Sdk",
    "command": "hello.sh",
    "finish-args" : [
        "--share=network",
        "--share=ipc",
        "--socket=fallback-x11",
        "--socket=wayland"
    ],
    "modules": [
        {
            "name": "my_first_dep-face_recognition",
            "buildsystem": "simple",
            "build-commands": [
                "/usr/bin/pip3 install face_recognition --prefix=/app --verbose"
            ],
            "build-options": {
              "build-args": [
                "--share=network"
              ]
            }
        },
        {
            "name": "hello",
            "buildsystem": "simple",
            "build-commands": [
                "install -D hello.sh /app/bin/hello.sh"
            ],
            "sources": [
                {
                    "type": "file",
                    "path": "hello.sh"
                }
            ]
        }
    ]
}

It can be run with:

flatpak-builder --install --user build-dir com.felixbrezo.FlatpakTest.json --force-clean

Note that the dependencies for running Flatpak are:

sudo apt install flatpak flatpak-builder

The result is this one:

  [ 77%] Building CXX object CMakeFiles/dlib_python.dir/src/basic.cpp.o
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/matrix.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/svm_rank_trainer.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:96: CMakeFiles/dlib_python.dir/src/matrix.cpp.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:135: CMakeFiles/dlib_python.dir/src/svm_rank_trainer.cpp.o] Error 1
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/dlib.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:83: CMakeFiles/dlib_python.dir/src/dlib.cpp.o] Error 1
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/basic.cpp:3:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/other.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/vector.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:161: CMakeFiles/dlib_python.dir/src/other.cpp.o] Error 1
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/decision_functions.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:109: CMakeFiles/dlib_python.dir/src/vector.cpp.o] Error 1
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:148: CMakeFiles/dlib_python.dir/src/decision_functions.cpp.o] Error 1
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:174: CMakeFiles/dlib_python.dir/src/basic.cpp.o] Error 1
    In file included from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-smfahopz/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/opaque_types.h:6,
                     from /tmp/pip-install-smfahopz/dlib/tools/python/src/svm_c_trainer.cpp:4:
    /tmp/pip-install-smfahopz/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111:10: fatal error: Python.h: No such file or directory
      111 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    make[2]: *** [CMakeFiles/dlib_python.dir/build.make:122: CMakeFiles/dlib_python.dir/src/svm_c_trainer.cpp.o] Error 1
    make[1]: *** [CMakeFiles/Makefile2:137: CMakeFiles/dlib_python.dir/all] Error 2
    make: *** [Makefile:104: all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-smfahopz/dlib/setup.py", line 261, in <module>
        'Topic :: Software Development',
      File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-smfahopz/dlib/setup.py", line 135, in run
        self.build_extension(ext)
      File "/tmp/pip-install-smfahopz/dlib/setup.py", line 175, in build_extension
        subprocess.check_call(cmake_build, cwd=build_folder)
      File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j8']' returned non-zero exit status 2.
  Running setup.py install for dlib ... error
Cleaning up...
  Removing source in /tmp/pip-install-smfahopz/dlib
Removed build tracker '/tmp/pip-req-tracker-frsm_lmj'
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-smfahopz/dlib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-smfahopz/dlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-nbbbceqn/install-record.txt --single-version-externally-managed --compile --user --prefix= Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/usr/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/usr/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    **kwargs
  File "/usr/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 959, in install
    spinner=spinner,
  File "/usr/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 931, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-smfahopz/dlib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-smfahopz/dlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-nbbbceqn/install-record.txt --single-version-externally-managed --compile --user --prefix= Check the logs for full command output.

Things which are important to know:

• Flatpak makes the /app path writable.
• The simple build system is a typical way of fixing pip dependencies. More info about why this is important here.
• This code runs perfectly, for instance, to install requests module.
• I may be missing python-dev-tools, but also experimenting issues when installing them before face_recognition in this case with a OSError as follows:

OSError: [Errno 30] Read-only file system: '/usr/bin/pip3'

[Sternagfonkel]

Facing the same problem here. Any news on this?