meta-browser
meta-browser copied to clipboard
OSSystems
Firefox fails to build due to missing python module
Hi I'm a bit new to firefox/rust, but it fails to build for armv7.
------
0:11.06 Compiling webrender v0.60.0 (/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/gfx/wr/webrender)
0:11.08 error: failed to run custom build command for `style v0.0.1 (/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/servo/components/style)`
0:11.08 Caused by:
0:11.08 process didn't exit successfully: `/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/release/build/style-01cf4f6b28ee30c3/build-script-build` (exit code: 1)
------
0:11.08 --- stderr
0:11.08 Could not find platform independent libraries <prefix>
0:11.08 Could not find platform dependent libraries <exec_prefix>
0:11.08 Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
0:11.08 ImportError: No module named site
0:11.08 warning: build failed, waiting for other jobs to finish..
------
4:43.96 error: build failed
4:43.96 make[4]: *** [/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/config/makefiles/rust.mk:240: force-cargo-library-build] Error 101
4:43.96 make[3]: *** [/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/config/recurse.mk:74: toolkit/library/rust/target] Error 2
4:43.96 make[3]: *** Waiting for unfinished jobs....
5:46.69 make[2]: *** [/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/config/recurse.mk:34: compile] Error 2
5:46.69 make[1]: *** [/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/config/rules.mk:413: default] Error 2
5:46.69 make: *** [client.mk:125: build] Error 2
It seems to be missing some python module "site" ? Any ideas?
@kraj unfortunately bitbake -e firefox doesn't expose PYTHONPATH or PYTHONHOME. However it does say:
# $PYTHON
# set /home/mastag/src/odroid-oe-core/openembedded-core/meta/conf/bitbake.conf:533
# "${@sys.executable}"
PYTHON="/usr/bin/python3"
So I can assume it's using python3 from my host? That one is capable of importing "site":
Python 3.7.5 (default, Oct 17 2019, 12:16:48)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
De compile log does say: 0:11.08 Consider setting $PYTHONHOME to
Could it be that it's unset by default?
interesting, I wonder if its defaulting to modules from python that yocto builds and python binary from host then it might be an issue where yocto python3-native might not be building site module.
I'm seeing a similar error in building Firefox:
| 15:36.62 error: build failed
| 15:36.63 make[4]: *** [/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/config/makefiles/rust.mk:240: force-cargo-library-build] Error 101
| 15:36.63 make[3]: *** [/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/config/recurse.mk:74: toolkit/library/rust/target] Error 2
| 15:36.63 make[3]: *** Waiting for unfinished jobs....
| 16:22.83 make[2]: *** [/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/config/recurse.mk:34: compile] Error 2
| 16:22.83 make[1]: *** [/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/config/rules.mk:413: default] Error 2
| 16:22.83 make: *** [client.mk:125: build] Error 2
And prior to that in the logs, the Python error:
| 13:45.45 --- stderr
| 13:45.45 Traceback (most recent call last):
| 13:45.45 File "/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/servo/components/style/properties/build.py", line 5, in <module>
| 13:45.45 import json
| 13:45.45 File "/usr/lib/python2.7/json/__init__.py", line 108, in <module>
| 13:45.45 from .decoder import JSONDecoder
| 13:45.45 File "/usr/lib/python2.7/json/decoder.py", line 5, in <module>
| 13:45.45 import struct
| 13:45.45 File "/usr/lib/python2.7/struct.py", line 1, in <module>
| 13:45.45 from _struct import *
| 13:45.45 ImportError: /usr/lib/python2.7/lib-dynload/_struct.so: undefined symbol: PyUnicodeUCS4_AsEncodedString
| 13:45.45 warning: build failed, waiting for other jobs to finish...
I can't reproduce it. Could you show me your build environment's information?
- Host OS, Version
- bblayers.conf and local.conf
In addition, please show me PYTHON in your config.status (tmp/work/<target-arch>/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/config.status)
@ashie Build is on Manjaro - kernel 4.19.88-1. All my recipes are from the Yocto zeus branch.
From config.status:
'PYTHON': '/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/_virtualenvs/init/bin/python',
'PYTHON3': '/var/tmp/yocto/hosttools/python3',
'PYTHON3_VERSION': '3.7.4',
And bblayers.conf:
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
${TOPDIR}/../meta \
${TOPDIR}/../meta-poky \
${TOPDIR}/../meta-yocto-bsp \
${TOPDIR}/../meta-openembedded/meta-filesystems \
${TOPDIR}/../meta-openembedded/meta-gnome \
${TOPDIR}/../meta-openembedded/meta-initramfs \
${TOPDIR}/../meta-openembedded/meta-multimedia \
${TOPDIR}/../meta-openembedded/meta-networking \
${TOPDIR}/../meta-openembedded/meta-oe \
${TOPDIR}/../meta-openembedded/meta-perl \
${TOPDIR}/../meta-openembedded/meta-python \
${TOPDIR}/../meta-openembedded/meta-webserver \
${TOPDIR}/../meta-openembedded/meta-xfce \
${TOPDIR}/../meta-browser \
${TOPDIR}/../meta-rust \
${TOPDIR}/../meta-intel \
${TOPDIR}/../meta-clang \
"
'PYTHON': '/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/_virtualenvs/init/bin/python',
Hmm, it's almost same with mine. Is it capable of importing "site"? (It's capable on my environment)
Manjaro
It's interesting, I've heard for the first time at this time. I don't have knowledge about Arch Linux based distributions, I'll try it when I have spare time...
@ashie No issues importing "site" with the Python version in the Firefox build directory or with my host python.
Another interesting note, when I run bitbake -e firefox and search for PYTHON I see this:
# $PYTHON
PYTHON="/usr/bin/python3"
Confirmed the issue in Manjaro and in Ubuntu:16.04 Docker image with x86 Target.
| 0:03.28 error: failed to run custom build command for `style v0.0.1 (/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/servo/components/style)`
| 0:03.28 Caused by:
| 0:03.28 process didn't exit successfully: `/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/release/build/style-01cf4f6b28ee30c3/build-script-build` (exit code: 1)
| 0:03.28 --- stdout
| 0:03.28 cargo:rerun-if-changed=build.rs
| 0:03.28 cargo:out_dir=/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/x86_64-poky-linux/release/build/style-209e7c87851a4395/out
| 0:03.29 cargo:rerun-if-changed=properties/helpers/animated_properties.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/gecko.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/properties.html.mako
| 0:03.29 cargo:rerun-if-changed=properties/computed_value_flags.rs
| 0:03.29 cargo:rerun-if-changed=properties/Mako-0.9.1.zip
| 0:03.29 cargo:rerun-if-changed=properties/longhands/padding.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/inherited_box.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/outline.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/effects.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/inherited_table.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/list.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/svg.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/xul.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/ui.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/inherited_text.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/background.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/text.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/margin.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/table.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/border.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/font.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/column.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/color.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/position.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/inherited_svg.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/box.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/counters.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/longhands/inherited_ui.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/build.py
| 0:03.29 cargo:rerun-if-changed=properties/declaration_block.rs
| 0:03.29 cargo:rerun-if-changed=properties/properties.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/data.py
| 0:03.29 cargo:rerun-if-changed=properties/helpers.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/padding.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/outline.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/list.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/svg.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/inherited_text.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/background.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/text.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/margin.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/border.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/font.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/column.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/position.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/inherited_svg.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/shorthands/box.mako.rs
| 0:03.29 cargo:rerun-if-changed=properties/cascade.rs
| 0:03.29 --- stderr
| 0:03.29 Traceback (most recent call last):
| 0:03.29 File "/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/firefox-68.0/servo/components/style/properties/build.py", line 5, in <module>
| 0:03.29 import json
| 0:03.29 File "/usr/lib/python2.7/json/__init__.py", line 108, in <module>
| 0:03.29 from .decoder import JSONDecoder
| 0:03.29 File "/usr/lib/python2.7/json/decoder.py", line 5, in <module>
| 0:03.29 import struct
| 0:03.29 File "/usr/lib/python2.7/struct.py", line 1, in <module>
| 0:03.29 from _struct import *
| 0:03.29 ImportError: /usr/lib/python2.7/lib-dynload/_struct.so: undefined symbol: PyUnicodeUCS4_AsEncodedString
| 0:03.29 warning: build failed, waiting for other jobs to finish...
One aspect to explore would be to see how python2 is built on host as well as python-native on yocto perhaps try to run following program with python2 from yocto build and the native one
import sys
print(sys.maxunicode)
From within the docker container:
yocto@138ad52962b6:/yocto$ python2
Python 2.7.12 (default, Oct 8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.maxunicode)
1114111
>>> exit()
yocto@138ad52962b6:/yocto$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Python seems to have no problem importing json from within the docker shell?
and what do you get when you run python2 from sysroot-native that yocto built ?
@kraj If you're referring to the python2 at /var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/bin/python-native, I get this:
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/bin/python-native ᐅ ./python2
Python 2.7.16 (default, Dec 10 2019, 23:39:41)
[GCC 9.1.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.maxunicode)
65535
@dswalla yes thats the problem. Yocto compiles its own version of python2 and it seems to be using modules from host, and they wont work reliably if the pythons are not compiled with same settings/features in this case we can see that your host python2 is compiled with 32bit encoding support but yocto version is compiled with 16-bit encoding support for unicode.
Interesting. Is there a portion of the Firefox build script that overrides the python modules path? Running the python2-native executable and checking the path produces the following:
~ ᐅ /var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/bin/python-native/python2
Python 2.7.16 (default, Dec 10 2019, 23:39:41)
[GCC 9.1.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print '\n'.join(sys.path)
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python27.zip
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/plat-linux2
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/lib-tk
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/lib-old
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/lib-dynload
/var/tmp/yocto/work/corei7-64-poky-linux/firefox/68.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/site-packages
@kraj with the latest openembedded-core, meta-openembedded and meta-python2 it seems to build now (still building so I'm not sure whether it will finish successfully), However I've had to create a .bbappend for firefox with: DEPENDS += "python-native" in order for it to detect python2.
EDIT: Well it failed again but not related to this issue.
4:06.22 Compiling audioipc v0.2.4 (/home/mastag/src/odroid-oe-core/build/tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/firefox-68.0/media/audioipc/audioipc)
4:06.24 error[E0506]: cannot assign to self.input.cached_token because it is borrowed
4:06.24 --> /usr/src/debug/firefox/68.0esr-r0/firefox-68.0/third_party/rust/cssparser/src/parser.rs:584:17
4:06.24 |
4:06.24 559 | pub fn next_including_whitespace_and_comments(&mut self) -> Result<&Token<'i>, BasicParseError<'i>> {
4:06.24 | - let's call the lifetime of this reference '1
4:06.24 ...
4:06.24 572 | Some(ref cached_token)
4:06.24 | ---------------- borrow of self.input.cached_token occurs here
4:06.24 ...
4:06.24 584 | self.input.cached_token = Some(CachedToken {
4:06.24 | ^^^^^^^^^^^^^^^^^^^^^^^ assignment to borrowed self.input.cached_token occurs here
4:06.24 ...
4:06.24 596 | Ok(token)
4:06.24 | --------- returning this value requires that self.input.cached_token.0 is borrowed for '1
4:06.26 8 warnings generated.
4:06.27 Compiling rkv v0.9.4
So I guess the python problems are solved when using upstream version of bitbake, openembedded-core, meta-openembedded and meta-python2. All it needs is a: DEPENDS += "python-native"
Small update, the python problem seems to persist. Firefox can still be built using:
PREFERRED_VERSION_rust-native ?= "1.37.0" PREFERRED_VERSION_rust-cross-aarch64 ?= "1.37.0" PREFERRED_VERSION_rust-llvm-native ?= "1.37.0" PREFERRED_VERSION_libstd-rs ?= "1.37.0" PREFERRED_VERSION_cargo-native ?= "1.37.0"
Even when adding DEPENDS += "python-native" it seems to error out with the same message as my opening post:
2:02.38 Could not find platform independent libraries
When I run the native python I can import site without problems: ./tmp/work/cortexa15t2hf-neon-vfpv4-oe-linux-gnueabi/firefox/68.0esr-r0/recipe-sysroot-native/usr/bin/python-native/python Python 2.7.17 (default, Mar 10 2020, 14:40:54) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import site
error[E0506]: cannot assign to self.input.cached_token because it is borrowed
The fix for this is available here: https://hg.mozilla.org/releases/mozilla-esr68/rev/f0020f517832
I've already backported it to meta-browser in: https://github.com/shr-project/meta-browser/commits/jansa/rust but then it panics elsewhere in rust and I haven't found a fix for that yet (log.do_compile shown in the same branch).
For python2 issue in dunfell, even using pythonnative didn't work for me, only adding python to HOSTTOOLS worked for me on Ubuntu 20.04, I've added my WIP changes to: https://github.com/shr-project/meta-browser/commits/jansa/python
@shr-project I do guess that you are using zeus YPRR ? If yes, you would use gcc 9.x ?
@lmajewski no zeus, I'm using dunfell as mentioned with python2. So I'm using gcc 10.1.
For zeus it builds fine (with older meta-rust 5d1ada0c97723e1526bf5599b2fa2cbb56c2c0dc)
For python2 issue in dunfell, even using pythonnative didn't work for me, only adding python to HOSTTOOLS worked for me on Ubuntu 20.04,
I've confirmed it with dunfell & meta-python2. The python2.7 command is placed under:
- tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/recipe-sysroot-native/usr/bin/python-native/
and PATH is not set for it while most other commands are placed under:
- tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/recipe-sysroot-native/usr/bin
arent we using inherit pythonnative
We currently aren't and adding it doesn't work.
arent we using inherit pythonnative
Ah, I got it. meta-python2/classes/pythonnative.bbclass provides it.
EXTRANATIVEPATH += "python-native"
DEPENDS_append = " python-native "
arent we using inherit pythonnative
It's still failing with the following error:
| 0:00.34 File "/build/master/build/tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/recipe-sysroot-native/usr/lib/python2.7/distutils/sysconfig.py", line 424, in _init_posix
| 0:00.34 raise DistutilsPlatformError(my_msg)
| 0:00.34 distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /build/master/build/tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/recipe-sysroot/usr/lib/python2.7/config/Makefile (No such file or directory)
It should be recipe-sysroot-native?
It probably should be recipe-sysroot-native, because even if you add target python to resolve this issue it will try to use the target library from there and fail on host
0:01.82 /OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/firefox/68.0esr-r0/firefox-68.0/firefox-build-dir/_virtualenvs/init/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
as shown in top 2 commits of https://github.com/shr-project/meta-browser/commits/jansa/python
meta-python2/classes/pythonnative.bbclass:
# python-config and other scripts are using distutils modules
# which we patch to access these variables
export STAGING_INCDIR
export STAGING_LIBDIR
Probably it should be like this
export STAGING_INCDIR="${STAGING_INCDIR_NATIVE}"
export STAGING_LIBDIR="${STAGING_LIBDIR_NATIVE}"
The former one cannot run python-config but the later one can do it.
But I still got another error:
| 0:00.34 IOError: [Errno 13] Permission denied: '/build/master/build/tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/firefox-68.9.0/firefox-build-dir/
_virtualenvs/init/lib/python2.7/lib-dynload/readline.so'
It points system's one
$ LANG=C ls -l /build/master/build/tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/firefox-68.9.0/firefox-build-dir/_virtualenvs/init/lib/python2.7/lib-dynload
lrwxrwxrwx 1 aho aho 30 Jun 4 08:28 /build/master/build/tmp/work/core2-64-poky-linux/firefox/68.9.0esr-r0/firefox-68.9.0/firefox-build-dir/_virtualenvs/init/lib/python2.7/lib-dynload -> /usr/lib/python2.7/lib-dynload