Failure to import yum in CentOS7 bootstrap root, when /var/lib/mock/ is a bind-mount or symlink

[ydirson]

Short description of the problem

I can run mock --init to create a custom CentOS7-based rootfs, with the default layout where /var/lib/mock/ is a normal dir (which happen to live in the root partition).

Now I want to move this tree (as well as /var/cache/mock) on another partition. But when I use symlinks or bind mounts, mock --init fails with:

ERROR: Command failed: 
 # /usr/bin/yum --installroot /var/lib/mock/centos-7-x86_64/root/ --releasever 7 install bash bzip2 coreutils cpio diffutils findutils gawk gcc gcc-c++ grep gzip info make patch redhat-rpm-config rpm-build sed shadow-utils tar unzip util-linux which xz
error: Failed to initialize NSS library
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   cannot import name ts

This is the error we get when manually asking python to import yum in the bootstrap chroot, when /dev/ is not bind-mounted, and which disappears as soon as /dev/ is bind-mounted. So I guess mock for some reason would not setup this mountpoint /var/lib/mock/centos-7-x86_64/result/root.log does in fact not show it mount /dev, whereas it does mount /proc, but then it does not show it either when the run succeeds.

Output of rpm -q mock

mock-3.1-1.fc36.noarch

Steps to reproduce issue

1. sudo mkdir $HOME/mock/lib
2. sudo chgrp mock $HOME/mock/lib
3. sudo mount -o bind $HOME/mock/lib /var/lib/mock
4. mock -r centos-7-x86_64 --init

Any additional notes

Output of `mock --debug-config`

INFO: mock.py version 3.1 starting (python version = 3.10.7, NVR = mock-3.1-1.fc36)... Start(bootstrap): init plugins INFO: selinux disabled Finish(bootstrap): init plugins Start: init plugins INFO: selinux disabled Finish: init plugins INFO: Signal handler active Start: run config_opts['additional_packages'] = [] config_opts['bootstrap_image'] = 'registry.fedoraproject.org/fedora:{{ releasever }}' config_opts['build_log_fmt_str'] = '%(message)s' config_opts['cache_alterations'] = False config_opts['chroot_name'] = 'default' config_opts['chroot_setup_cmd'] = 'install @{% if mirrored %}buildsys-{% endif %}build' config_opts['cleanup_on_failure'] = False config_opts['cleanup_on_success'] = False config_opts['config_file'] = '/etc/mock/default.cfg' config_opts['config_path'] = '/etc/mock' config_opts['config_paths'] = ['/etc/mock/site-defaults.cfg', '/etc/mock/chroot-aliases.cfg', '/etc/mock/fedora-36-x86_64.cfg', '/etc/mock/templates/fedora-branched.tpl'] config_opts['description'] = 'Fedora {{ releasever }}' config_opts['dist'] = 'fc{{ releasever }}' config_opts['dnf.conf'] = ('\n' '[main]\n' 'keepcache=1\n' 'debuglevel=2\n' 'reposdir=/dev/null\n' 'logfile=/var/log/yum.log\n' 'retries=20\n' 'obsoletes=1\n' 'gpgcheck=0\n' 'assumeyes=1\n' 'syslog_ident=mock\n' 'syslog_device=\n' 'install_weak_deps=0\n' 'metadata_expire=0\n' 'best=1\n' 'module_platform_id=platform:f{{ releasever }}\n' 'protected_packages=\n' 'user_agent={{ user_agent }}\n' '\n' '# repos\n' '\n' '[local]\n' 'name=local\n' 'baseurl=https://kojipkgs.fedoraproject.org/repos/f{{ releasever ' '}}-build/latest/$basearch/\n' 'cost=2000\n' 'enabled={{ not mirrored }}\n' 'skip_if_unavailable=False\n' '\n' '{% if mirrored %}\n' '[fedora]\n' 'name=fedora\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates]\n' 'name=updates\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-testing]\n' 'name=updates-testing\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[fedora-debuginfo]\n' 'name=fedora-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-debuginfo]\n' 'name=updates-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-debug-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-testing-debuginfo]\n' 'name=updates-testing-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-debug-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[fedora-source]\n' 'name=fedora-source\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'enabled=0\n' 'skip_if_unavailable=False\n' '\n' '[updates-source]\n' 'name=updates-source\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-source-f$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'enabled=0\n' 'skip_if_unavailable=False\n' '\n' '# modular\n' '\n' '[fedora-modular]\n' 'name=Fedora Modular $releasever - $basearch\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-$releasever&arch=$basearch\n' '# if you want to enable it, you should set best=0\n' '# see https://bugzilla.redhat.com/show_bug.cgi?id=1673851\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-$releasever-primary\n' 'skip_if_unavailable=False\n' '\n' '[fedora-modular-debuginfo]\n' 'name=Fedora Modular $releasever - $basearch - Debug\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-debug-$releasever&arch=$basearch\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-$releasever-primary\n' 'skip_if_unavailable=False\n' '\n' '[fedora-modular-source]\n' 'name=Fedora Modular $releasever - Source\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-source-$releasever&arch=$basearch\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-$releasever-primary\n' 'skip_if_unavailable=False\n' '\n' '[updates-modular]\n' 'name=Fedora Modular $releasever - $basearch - Updates\n' '#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/Modular/$basearch/\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f$releasever&arch=$basearch\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch\n' 'skip_if_unavailable=False\n' '\n' '[updates-modular-debuginfo]\n' 'name=Fedora Modular $releasever - $basearch - Updates - Debug\n' '#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/Modular/$basearch/debug/\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-debug-f$releasever&arch=$basearch\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch\n' 'skip_if_unavailable=False\n' '\n' '[updates-modular-source]\n' 'name=Fedora Modular $releasever - Updates Source\n' '#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/Modular/SRPMS/\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-source-f$releasever&arch=$basearch\n' 'enabled=0\n' 'repo_gpgcheck=0\n' 'type=rpm\n' 'gpgcheck=1\n' 'gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch\n' 'skip_if_unavailable=False\n' '{% endif %}\n') config_opts['enable_disable_repos'] = [] config_opts['extra_chroot_dirs'] = ['/run/lock'] config_opts['files'] = {'etc/hosts': '127.0.0.1 localhost localhost.localdomain\n' '::1 localhost localhost.localdomain localhost6 ' 'localhost6.localdomain6\n'} config_opts['legal_host_arches'] = ('x86_64',) config_opts['macros'] = {'%_buildhost': 'localhost.localdomain', '%_platform_multiplier': 1, '%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm', '%_topdir': '/builddir/build'} config_opts['mirrored'] = True config_opts['no-config'] = {'epel-8': {'alternatives': {'alma+epel-8': {'description': ['Builds against ' 'AlmaLinux 8 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://almalinux.org/']}, 'centos-stream+epel-8': {'description': ['Builds ' 'against ' 'CentOS ' 'Stream ' '8 ' 'repositories ' '(some ' 'packages ' 'may be ' 'a bit ' 'ahead ' 'the Red ' 'Hat ' 'Enterprise ' 'Linux ' '8) ' 'together ' 'with ' 'the ' 'official ' 'EPEL 8 ' 'repositories.', 'Project ' 'page: ' 'https://www.centos.org/centos-stream/']}, 'circlelinux+epel-8': {'description': ['Builds ' 'against ' 'Circle ' 'Linux 8 ' 'repositories, ' 'together ' 'with the ' 'official ' 'EPEL ' 'repositories.', 'Project ' 'page: ' 'https://cclinux.org/']}, 'rhel+epel-8': {'description': ['Builds against ' 'Red Hat ' 'Enterprise Linux ' '8 repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'This mimics what ' 'is done in the ' 'official EPEL ' 'build system, ' 'but you need a ' 'Red Hat ' 'subscription:', 'https://rpm-software-management.github.io/mock/Feature-rhelchroots']}, 'rocky+epel-8': {'description': ['Builds against ' 'Rocky Linux 8 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://rockylinux.org/']}}}, 'epel-9': {'alternatives': {'alma+epel-9': {'description': ['Builds against ' 'AlmaLinux 9 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://almalinux.org/']}, 'centos-stream+epel-9': {'description': ['Builds ' 'against ' 'CentOS ' 'Stream ' '9 ' 'repositories ' '(some ' 'packages ' 'may be ' 'a bit ' 'ahead ' 'the Red ' 'Hat ' 'Enterprise ' 'Linux ' '9) ' 'together ' 'with ' 'the ' 'official ' 'EPEL ' 'repositories.', 'Project ' 'page: ' 'https://www.centos.org/centos-stream/']}, 'rhel+epel-9': {'description': ['Builds against ' 'Red Hat ' 'Enterprise Linux ' '9 repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'This mimics what ' 'is done in the ' 'official EPEL ' 'build system, ' 'but you need a ' 'Red Hat ' 'subscription:', 'https://rpm-software-management.github.io/mock/Feature-rhelchroots']}, 'rocky+epel-9': {'description': ['Builds against ' 'Rocky Linux 9 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://rockylinux.org/']}}}} config_opts['nspawn_args'] = ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.s7mmkfbb:/etc/resolv.conf'] config_opts['plugin_conf'] = {'bind_mount_enable': True, 'bind_mount_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'create_dirs': False, 'dirs': [], 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'ccache_enable': False, 'ccache_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'compress': None, 'dir': '/var/cache/mock/fedora-36-x86_64/ccache/u1000/', 'max_cache_size': '4G', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'chroot_scan_enable': False, 'chroot_scan_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'only_failed': True, 'regexes': ['^[^k]?core(\.\d+)?$', '\.log$'], 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'compress_logs_enable': False, 'compress_logs_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'command': 'gzip', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'hw_info_enable': True, 'hw_info_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'lvm_root_enable': False, 'lvm_root_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'pool_name': 'mockbuild', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'mount_enable': True, 'mount_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'dirs': [], 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'overlayfs_enable': False, 'overlayfs_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'package_state_enable': True, 'package_state_opts': {'available_pkgs': False, 'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'installed_pkgs': True, 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'pm_request_enable': False, 'pm_request_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'procenv_enable': False, 'procenv_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'root_cache_enable': True, 'root_cache_opts': {'age_check': True, 'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'compress_program': 'pigz', 'decompress_program': None, 'dir': '/var/cache/mock/fedora-36-x86_64/root_cache/', 'exclude_dirs': ['./proc', './sys', './dev', './tmp/ccache', './var/cache/yum', './var/cache/dnf', './var/log'], 'extension': '.gz', 'max_age_days': 15, 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'rpkg_preprocessor_enable': False, 'rpkg_preprocessor_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'cmd': '/usr/bin/preproc-rpmspec %(source_spec)s ' '--output %(target_spec)s', 'requires': ['preproc-rpmspec'], 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'selinux_enable': True, 'selinux_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'showrc_enable': False, 'showrc_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'sign_enable': False, 'sign_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'cmd': 'rpmsign', 'opts': '--addsign %(rpms)s', 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'tmpfs_enable': False, 'tmpfs_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'keep_mounted': False, 'max_fs_size': None, 'mode': '0755', 'required_ram_mb': 900, 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}, 'yum_cache_enable': True, 'yum_cache_opts': {'basedir': '/var/lib/mock/fedora-36-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-36-x86_64', 'max_age_days': 30, 'max_metadata_age_days': 30, 'online': True, 'resultdir': '/var/lib/mock/fedora-36-x86_64/result', 'root': 'fedora-36-x86_64'}} config_opts['print_main_output'] = True config_opts['releasever'] = '36' config_opts['root'] = 'fedora-{{ releasever }}-{{ target_arch }}' config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d: %(message)s' config_opts['rpmbuild_arch'] = 'x86_64' config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s' config_opts['target_arch'] = 'x86_64' config_opts['verbose'] = 1 Finish: run

[praiskup]

Thank you for the report. Please make sure the '$HOME/mock/lib' and $HOME/mock is 0775. Can you re-test?

[ydirson]

$HOME/mock was 0755 and $HOME/mock/lib was 2775 (was moved from /var/lib):

$ ls -ld ~/mock/lib/
drwxrwsr-x 6 root mock 4096 Oct  4 16:45 /home/user/mock/lib/
$ ls -ld ~/mock/
drwxr-xr-x 4 root mock 4096 Oct  4 14:28 /home/user/mock/

Setting both to 0755 (taking care of clearing the s bit explicitly) does not change the behavior.

Are you able to reproduce in any way ?

[praiskup]

Please try 0775. I can't reproduce if mock group has write access there.

[ydirson]

0775 does not change anything. That does not surprise me much: at the point the error occurs, the filesystem has already be (at least in large part) initialized:

$ sudo du -sh ~/mock/lib/*
6.3M	/home/user/mock/lib/centos-7-x86_64
292M	/home/user/mock/lib/centos-7-x86_64-bootstrap

[praiskup]

Hmm, sorry, I still can not reproduce this. When mock creates those directories, it creates them so mock group has access there (mock group ownership). Is this still an issue?

[ydirson]

We finally decided that for our needs (generating a rootfs) using Mock was not the best way (we went to direct yum usage, and may end up with directly calling rpm), so that issue is not really a problem for us any more.

[praiskup]

I see, no problem. Thank you for the question anyway!