python3/host: build install race condition with uboot/scripts/dtc/pylibfdt

[ynezz]

Maintainer: @jefferyto Environment: imx/cortexa9, OpenWrt buildbot for snapshots

Description:

We're experiencing following occasional build failures:

  DTC     arch/arm/dts/imx6dl-mamoj.dtb
make[3]: Entering directory '/builder/shared-workdir/build/build_dir/hostpkg/Python-3.10.5'
...snip...
/builder/shared-workdir/build/staging_dir/host/bin/install -c python /builder/shared-workdir/build/staging_dir/hostpkg/bin/python3.10
...snip...
(cd /builder/shared-workdir/build/staging_dir/hostpkg/bin; ln -s python3.10 python3)
...snip...
  DTOC    spl/dts/dt-plat.c
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/asyncio/sslproto.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/asyncio
...snip...
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = 'scripts/dtc/pylibfdt'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/builder/shared-workdir/build/staging_dir/hostpkg/bin/python3'
  sys.base_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.base_exec_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.platlibdir = 'lib'
  sys.executable = '/builder/shared-workdir/build/staging_dir/hostpkg/bin/python3'
  sys.prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.exec_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.path = [
    'scripts/dtc/pylibfdt',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python310.zip',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: no codec search functions registered: can't find encoding

Current thread 0x00007f8429233740 (most recent call first):
  <no Python frame>
make[4]: *** [scripts/Makefile.spl:355: spl/dts/dt-plat.c] Error 1
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/asyncio/trsock.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/asyncio
make[3]: *** [Makefile:2012: spl/u-boot-spl] Error 2
make[3]: Leaving directory '/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01'
...snip...
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/encodings/utf_8.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/encodings

It looks like python3/host should make this symlink /builder/shared-workdir/build/staging_dir/hostpkg/bin/python3 available once the installation of python3/host is complete, not sooner, otherwise some Python3 scripts might start using incomplete Python3 interpreter installation which likely results in this failure.

[ynezz]

Another one:

/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/encodings/shift_jis_2004.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/encodings
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = 'scripts/dtc/pylibfdt'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/builder/shared-workdir/build/staging_dir/hostpkg/bin/python3'
  sys.base_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.base_exec_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.platlibdir = 'lib' 
  sys.executable = '/builder/shared-workdir/build/staging_dir/hostpkg/bin/python3'
  sys.prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.exec_prefix = '/builder/shared-workdir/build/staging_dir/hostpkg'
  sys.path = [
    'scripts/dtc/pylibfdt',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python310.zip',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10',
    '/builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: utf-8 

Current thread 0x00007fc2d9cb0740 (most recent call first):
  <no Python frame>
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/encodings/shift_jisx0213.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/encodings
make[4]: *** [scripts/Makefile.spl:355: include/generated/dt-decl.h] Error 1
make[4]: *** Deleting file 'include/generated/dt-decl.h'

[ynezz]

Another one:

/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/ensurepip/_bundle
  COPY    spl/u-boot-spl.dtb
  DTOC    spl/dts/dt-plat.c
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/html/__init__.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/html
...
builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/idlelib/statusbar.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/idlelib
/builder/shared-wo
rkdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/idlelib/textview.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/idlelib

make[1]: *** [package/Makefile:119: package/boot/uboot-imx/compile] Error 1
make[1]: *** Waiting for unfinished jobs....
...
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/lib2to3/fixes/fix_long.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/lib2to3/fixes
Traceback (most recent call last):
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/./tools/dtoc/dtoc", line 38, in <module>
    from dtoc import dtb_platdata
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/tools/dtoc/../dtoc/dtb_platdata.py", line 23, in <module>
    from dtoc import fdt
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/tools/dtoc/../dtoc/fdt.py", line 12, in <module>
    from dtoc import fdt_util
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/tools/dtoc/../dtoc/fdt_util.py", line 16, in <module>
    from patman import command
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/tools/dtoc/../patman/command.py", line 7, in <module>
    from patman import cros_subprocess
  File "/builder/shared-workdir/build/build_dir/target-arm_cortex-a9+neon_musl_eabi/u-boot-mx6cuboxi/u-boot-2022.01/tools/dtoc/../patman/cros_subprocess.py", line 262, in <module>
    class TestSubprocess(unittest.TestCase):
AttributeError: module 'unittest' has no attribute 'TestCase'
/builder/shared-workdir/build/staging_dir/host/bin/install -c -m 644 ./Lib/lib2to3/fixes/fix_map.py /builder/shared-workdir/build/staging_dir/hostpkg/lib/python3.10/lib2to3/fixes
...
make[4]: *** [scripts/Makefile.spl:355: spl/dts/dt-plat.c] Error 1