pkgdev manifest doesn't die on (some?) global-scope failures

[thesamesam]

@arthurzam mentioned this issue the other say in #gentoo-dev but I figured I should file a bug for it the next time I hit it.

I typo'd inherit bash-completion instead of inherit bash-completion-r1 and pkgdev manifest gave me nothing, instead of erroring on the failure in global scope (inheriting a non-existent eclass):

~/git/overlay/dev-util/meson $ head -n20 meson-0.61.4-r2.ebuild 
# Copyright 2016-2022 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

PYTHON_COMPAT=( python3_{8,9,10} )
DISTUTILS_USE_PEP517=setuptools

if [[ ${PV} = *9999* ]]; then
        EGIT_REPO_URI="https://github.com/mesonbuild/meson"
        inherit git-r3
else
        SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.tar.gz"
        KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86 ~x64-cygwin ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~sparc-solaris ~sparc64-solaris ~x64-solaris ~x86-solaris"
fi

inherit bash-completion distutils-r1 toolchain-funcs

DESCRIPTION="Open source build system"
HOMEPAGE="https://mesonbuild.com/"

~/git/overlay/dev-util/meson $ pkgdev manifest
# no output

~/git/overlay/dev-util/meson $ ebuild meson-0.61.4-r2.ebuild clean install
 * ERROR: dev-util/meson-0.61.4-r2::sam_c failed (depend phase):
 *   bash-completion.eclass could not be found by inherit()
 * 
 * Call stack:
 *                ebuild.sh, line 618:  Called source '/home/sjames/git/overlay/dev-util/meson/meson-0.61.4-r2.ebuild'
 *   meson-0.61.4-r2.ebuild, line  17:  Called inherit 'bash-completion' 'distutils-r1' 'toolchain-funcs'
 *                ebuild.sh, line 264:  Called die
 * The specific snippet of code:
 *              [[ -z ${location} ]] && die "${1}.eclass could not be found by inherit()"
 * 
 * If you need support, post the output of `emerge --info '=dev-util/meson-0.61.4-r2::sam_c'`,
 * the complete build log and the output of `emerge -pqv '=dev-util/meson-0.61.4-r2::sam_c'`.
 * Working directory: '/usr/lib/python3.10/site-packages'
 * S: '/var/tmp/portage/dev-util/meson-0.61.4-r2/work/meson-0.61.4'
~/git/overlay/dev-util/meson $ 

[thesamesam]

Oh, weird. meson-9999.ebuild was identical, but when I fixed 0.61.4 (and not 9999), it started to fail!

 ~/git/overlay/dev-util/meson $ pkgdev manifest
 * dev-util/meson-9999: failed sourcing ebuild: inherit requires unknown eclass: bash-completion.eclass

[radhermit]

Modifying two ebuilds with nonexistent eclass inherits shows the correct output for me so I'm not entirely sure what's happening:

 * dev-util/meson-9999: failed sourcing ebuild: inherit requires unknown eclass: bash-completion.eclass
 * dev-util/meson-0.60.2-r1: failed sourcing ebuild: inherit requires unknown eclass: bash-completion.eclass

[zhuyifei1999]

The root cause is here:

https://github.com/pkgcore/pkgcore/blob/master/src/pkgcore/ebuild/repository.py#L58-L69

        for pkgs in map(list, pkgutils.groupby_pkg(matches)):
            key = pkgs[0].key
            manifest = pkgs[0].manifest

            # check for pkgs masked by bad metadata
            if bad_metadata := self.repo._bad_masked.match(pkgs[0].unversioned_atom):
                for pkg in bad_metadata:
                    exc = pkg.data
                    error_str = f"{pkg.cpvstr}: {exc.msg(verbosity=observer.verbosity)}"
                    observer.error(error_str)
                    ret.add(pkg.key)
                continue

To see this, check out the stack trace of when the message is printed:

Traceback (most recent call first):
  File "snakeoil/formatters.py", line 387, in __call__
    formatter.stream.write(res)
  File "snakeoil/formatters.py", line 190, in _write_prefix
    thing = thing(self)
  File "snakeoil/formatters.py", line 245, in write
    self._write_prefix(wrap)
  File "snakeoil/formatters.py", line 503, in write
    super().write(*args, **kwargs)
  File "pkgcore/operations/observer.py", line 73, in error
    self._out.write(_convert(msg, args, kwds), prefixes=prefixes)
  File "pkgcore/ebuild/repository.py", line 67, in _cmd_implementation_manifest
    observer.error(error_str)
  File "pkgcore/operations/repo.py", line 283, in _cmd_api_manifest
    return self._cmd_implementation_manifest(
  File "pkgcore/operations/__init__.py", line 70, in _recast_exception_decorator
    return functor(*args, **kwds)
  File "pkgdev/scripts/pkgdev_manifest.py", line 129, in _manifest
    failed = options.repo.operations.manifest(
  File "snakeoil/cli/tool.py", line 176, in main
    exitstatus = func(self.options, self.out, self.err)
  File "pkgdev/cli.py", line 23, in main
    return super().main()
  File "snakeoil/cli/tool.py", line 81, in __call__
    ret = self.main()
  File "pkgdev/scripts/__init__.py", line 40, in run
    sys.exit(tool())
  File "pkgdev/scripts/__init__.py", line 48, in main
    run(os.path.basename(sys.argv[0]))
  File "bin/pkgdev", line 8, in <module>
    sys.exit(main())

And of when it kills the ebuild (and raise the exception):

Traceback (most recent call first):
  <built-in method killpg of module object at remote 0x7ffff75d5df0>
  File "pkgcore/ebuild/processor.py", line 724, in shutdown_processor
    os.killpg(self.pid, signal.SIGKILL)
  File "pkgcore/ebuild/processor.py", line 998, in inherit_handler
    ebp.shutdown_processor(force=True)
  File "pkgcore/ebuild/processor.py", line 973, in generic_handler
    handlers[cmd](self, *args)
  File "pkgcore/ebuild/processor.py", line 838, in _run_depend_like_phase
    self.generic_handler(additional_commands=commands)
  File "pkgcore/ebuild/processor.py", line 903, in get_keys
    self._run_depend_like_phase(
  File "pkgcore/ebuild/ebuild_src.py", line 525, in _update_metadata
    mydata = my_proc.get_keys(pkg, self._ecache)
  File "pkgcore/ebuild/ebuild_src.py", line 516, in _get_metadata
    return self._update_metadata(pkg, ebp=ebp)
  File "pkgcore/ebuild/ebuild_src.py", line 423, in _fetch_metadata
    return self._parent._get_metadata(self, ebp=ebp, force_regen=force_regen)
  File "pkgcore/package/metadata.py", line 56, in data
    return self._fetch_metadata()
  File "pkgcore/package/base.py", line 95, in dynamic_getattr_dict
    val = functor(self)
  File "pkgcore/ebuild/repository.py", line 585, in _pkg_filter
    pkg.data
  <built-in method sorted of module object at remote 0x7ffff75b0a90>
  File "pkgcore/repository/prototype.py", line 309, in _internal_gen_candidates
    yield from sorter(pkg_filter(pkgs))
  File "pkgcore/repository/prototype.py", line 314, in _internal_match
    for pkg in self._internal_gen_candidates(candidates, **kwargs):
  File "pkgcore/util/packages.py", line 15, in groupby_pkg
    for key, pkgs in itertools.groupby(iterable, attrgetter("key")):
  File "pkgcore/ebuild/repository.py", line 58, in _cmd_implementation_manifest
    for pkgs in map(list, pkgutils.groupby_pkg(matches)):
  File "pkgcore/operations/repo.py", line 283, in _cmd_api_manifest
    return self._cmd_implementation_manifest(
  File "pkgcore/operations/__init__.py", line 70, in _recast_exception_decorator
    return functor(*args, **kwds)
  File "pkgdev/scripts/pkgdev_manifest.py", line 129, in _manifest
    failed = options.repo.operations.manifest(
  File "snakeoil/cli/tool.py", line 176, in main
    exitstatus = func(self.options, self.out, self.err)
  File "pkgdev/cli.py", line 23, in main
    return super().main()
  File "snakeoil/cli/tool.py", line 81, in __call__
    ret = self.main()
  File "pkgdev/scripts/__init__.py", line 40, in run
    sys.exit(tool())
  File "pkgdev/scripts/__init__.py", line 48, in main
    run(os.path.basename(sys.argv[0]))
  File "bin/pkgdev", line 8, in <module>
    sys.exit(main())

The common ancestor is in the File "pkgcore/ebuild/repository.py", in _cmd_implementation_manifest, where

        for pkgs in map(list, pkgutils.groupby_pkg(matches)):  # <= raises error
            key = pkgs[0].key
            manifest = pkgs[0].manifest

            # check for pkgs masked by bad metadata
            if bad_metadata := self.repo._bad_masked.match(pkgs[0].unversioned_atom):
                for pkg in bad_metadata:
                    exc = pkg.data
                    error_str = f"{pkg.cpvstr}: {exc.msg(verbosity=observer.verbosity)}"
                    observer.error(error_str)  # <= prints error
                    ret.add(pkg.key)
                continue

Let's look at what map(list, pkgutils.groupby_pkg(matches)) iterates, and what bad_metadata is, when it prints:

map(list, pkgutils.groupby_pkg(matches)) =

[[<<class 'pkgcore.ebuild.ebuild_src.package'> cpv='dev-util/meson-1.0.1' @0x7f0c6ba39e40>, <<class 'pkgcore.ebuild.ebuild_src.package'> cpv='dev-util/meson-1.1.0' @0x7f0c6ba39bc0>]]

bad_metadata =

[<<class 'pkgcore.repository.virtual.InjectedPkg'> cpv='dev-util/meson-9999' @0x7f0c6ba7a560>]

i.e. the former lists the good ebuilds, the latter lists bad ebuilds:

gentoo/dev-util/meson $ grep '^inherit' *.ebuild
meson-1.0.1.ebuild:inherit bash-completion-r1 distutils-r1 toolchain-funcs
meson-1.1.0.ebuild:inherit bash-completion-r1 distutils-r1 toolchain-funcs
meson-9999.ebuild:inherit bash-completion distutils-r1 toolchain-funcs
gentoo/dev-util/meson $ pkgdev manifest
 * dev-util/meson-9999: failed sourcing ebuild: inherit requires unknown eclass: bash-completion.eclass

But what if all of them are bad?

gentoo/dev-util/meson $ sed -i 's/bash-completion-r1/bash-completion/' *.ebuild
gentoo/dev-util/meson $ grep '^inherit' *.ebuild
meson-1.0.1.ebuild:inherit bash-completion distutils-r1 toolchain-funcs
meson-1.1.0.ebuild:inherit bash-completion distutils-r1 toolchain-funcs
meson-9999.ebuild:inherit bash-completion distutils-r1 toolchain-funcs
gentoo/dev-util/meson $ pkgdev manifest

No output. There are no good ebuilds. The iterator does not yield anything. The output code path does not trigger.

[arthurzam]

@zhuyifei1999 Thank you very much. This reproducer works perfectly and the investigation is spot on.

This will help debugging, but the road is still hard, since I need to understand how to fix the issue. Maybe output the sourcing error even if can't improve other things? I'm looking at it.

[zhuyifei1999]

I need to understand how to fix the issue

If I knew how to fix this I would have sent a patch already :wink: I didn't because the bug looks structural and not some simple one-line change (or maybe it is but it's not obvious to me).

[echuber2]

After discussing the issue with @zhuyifei1999, we made a PR to the pkgcore repo that might fix this. It seems to work for the case described above and a few other contrived examples. https://github.com/pkgcore/pkgcore/pull/406

So it seems that if all ebuild files in a subdirectory get masked bad for whatever reason, the primary loop hides those errors and prevents exit 1 from happening. The itermatch here https://github.com/pkgcore/pkgcore/blob/83dd99d490fa4fd0785053b99ec617fd0851d409/src/pkgcore/ebuild/repository.py#L57 has the (implicit) default package filter set to skip bad masked files, but then the loop may not run for a package that is itself masked bad. The followup loop catches any masked-bad items that didn't already have their requesting package flagged as an error. (Someone please double-check if this is correct.) Maybe there's a more elegant solution, but I don't think it would otherwise be as simple as overriding the pkg_filter function passed to itermatch, because that breaks other things in the primary loop.

[thesamesam]

Thank you both @zhuyifei1999 and @echuber2 for the great investigation & fix!