koschei
koschei copied to clipboard
fedora-infra
Run builds from SCM instead of SRPM
There are test failures in the python2 package in Koschei:
https://apps.fedoraproject.org/koschei/package/python2
======================================================================
ERROR: test_lookup_issue1813 (test.test_codecs.CodecsModuleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-2.7.15/Lib/locale.py", line 581, in setlocale
return _setlocale(category, locale)
Error: unsupported locale setting
----------------------------------------------------------------------
======================================================================
ERROR: test_getsetlocale_issue1813 (test.test_locale.TestMiscellaneous)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-2.7.15/Lib/locale.py", line 581, in setlocale
return _setlocale(category, locale)
Error: unsupported locale setting
======================================================================
ERROR: test_setlocale_unicode (test.test_locale.TestMiscellaneous)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-2.7.15/Lib/locale.py", line 581, in setlocale
return _setlocale(category, locale)
Error: unsupported locale setting
----------------------------------------------------------------------
A attempted fix was at https://src.fedoraproject.org/rpms/python2/pull-request/32 - it built in the CI back then.
Examining the root logs from Koschei, glibc-all-langpacks is not installed.
Unfortunately Koschei does not list a dist-git commit hash, so I cannot tell whether this is some kind of resolver bug or if Koschei just builds old spec (that's what I suspect).
A manual scratchbuild installs glibc-all-langpacks and builds fine: https://koji.fedoraproject.org/koji/taskinfo?taskID=32017566
This behaviour is expected. From Koschei documentation on Fedora Wiki: "Packages are rebuilt in Koji from the latest available SRPM". The latest python2 package in f30-build is python2-2.7.15-10.fc30 (you can verify that with koji latest-pkg f30-build python2) and this version doesn't have BuildRequires on glibc-all-langpacks.
What do you propose as a solution? Submit scratch-builds from latest SCM commit?
Yes. So when a FTBFS starts happening, we don't need to bump release and build it to make it green in koschei. Just commit the fix. Also the other way around. If somebody commits a breakage, it should start failing. It will be built from scm after all once really built, not from latest SRPM.
The main issue with building packages from SCM is that this approach requires much more resources - in case of simple packages twice more resources compared to builds from existing SRPM. But I'm thinking that Koschei could give users a way to make particular packages built from SCM until a new real build is done. What do you think?
Opt-in works for me.
Could you please elaborate why does it need more resources?
Could you please elaborate why does it need more resources?
Probably because it would spawn 2 Koji jobs instead of 1, one for building a SRPM from SCM and the other one for doing the actual build.
However, I agree with @hroncok, this would be very helpful and a great improvement. Even if it is only an opt-in.
I'm fine with opt-in, but at the moment I don't have time to implement this RFE. Patches are welcome.
Since I have zero knowledge of this code base, I won't be able to send patches. Would you be able to show some early ideas (like "this needs to be changed here, the configuration is updated here") at Flock?
Since this has been left here without a reply, some examples of why I think this is actually needed to be done instead of SRPM rebuilds:
Green in koschei, but FTBFS:
https://koschei.fedoraproject.org/package/python-caja?collection=f32 https://bugzilla.redhat.com/show_bug.cgi?id=1792044
https://koschei.fedoraproject.org/package/otf2?collection=f32 https://bugzilla.redhat.com/show_bug.cgi?id=1791678
https://koschei.fedoraproject.org/package/jrnl?collection=f32 https://bugzilla.redhat.com/show_bug.cgi?id=1791686
https://koschei.fedoraproject.org/package/libgpiod?collection=f32 https://bugzilla.redhat.com/show_bug.cgi?id=1791985
Those are just several packages I found out about within a small window of going trough the Python 3.9 build failures. If koschei would build from git, we would have already known about those in advance.
My previous argument of SCM builds using more resources that SRPM builds is no longer valid - current Koji versions submit rebuildSRPM tasks that use amount of resources comparable with buildSRPMFromSCM, so I think this RFE can be implemented. I will try to get it done, but I'm currently busy with different work.
Implementation notes: Koschei doesn't have SCM knowledge any longer (branch info were removed in commits 91fad1e7fae83e91c3cb5d62374ca54d6bc855c2 and d3ec7c748dadfb701ace932250b7264339b88128), this information would need to be re-added. Then koschei.backend.submit_build() function can be simply changed to submit build from SCM if SCM info is specified in collection settings, and if not, fall back to SRPM build.
This feature was deployed to Fedora staging environment today and Koschei was submitting builds from SCM. Unfortunately blocker issues were discovered when testing in staging.
First, when submitting builds from SCM Koschei has no way of getting package BuildRequires for given build. BuildRequires are important information which Koschei needs for resolving package dependencies. For builds from pre-existing SRPM Koschei can get BuildRequires by querying SRPM headers. For SCM builds Koschei would need to download SRPM produced during scratch build and parse it instead. This functionality was removed long time ago in commit ae7d24d09ab10781fed1b029fcab7d4cf8994d98 and I don't want to add it at this moment.
Secondly, Koschei needs to know for which architectures to build particular package. When SRPM is available Koschei can use its headers (BUILDARCHS, EXCLUDEARCH, EXCLUSIVEARCH) to determine arches to build package for. For SCM build this becomes much more complex problem - arches need to be known before build is submitted, and at this point not even SRPM is available. Koschei would need to access SCM and parse spec file from there. Parsing spec files needs to be done in sandbox as spec files can execute arbitrary commands when they are parsed.
The above means that SCM builds won't work without further design and code changes and can't be enabled in production yet.
Koschei would need to access SCM and parse spec file from there.
That is the logical thing to do IMO.
Parsing spec files needs to be done in sandbox as spec files can execute arbitrary commands when they are parsed.
Hmm... could one avoid full spec file parsing if one only needs to extract BuildRequires and BuildArchs from the spec file?
Hmm... could one avoid full spec file parsing if one only needs to extract BuildRequires and BuildArchs from the spec file?
No, that is not possible in general.
Consider the following example:
%if %(/bin/echo 1)
BuildRequires: foo
BuildArch: x86_64
%endif
In order to parse this part of spec file you need to execute external command /bin/echo 1 in appropriate environment, which means chroot with exactly the same packages installed as in Koji buildroot.
I'm wondering why koji needs BRs/arches before building the SRPM, in other words: Why can't koji build the srpm (it rebuilds it now anyways) and then query it for arches and BRs? At worst, there is the following option:
- query the latest SRPM for info
- build SRPM and RPMs from SCM This might not even be bad but allow an additional check: compare the two SRPMs for essential equivalence and warn about differences (which are expected when SCM is ahead of the latest build, but not otherwise).
I'm wondering why koji needs BRs/arches before building the SRPM, in other words: Why can't koji build the srpm (it rebuilds it now anyways) and then query it for arches and BRs? At worst, there is the following option:
Koji doesn't need BRs and arches, Koschei does.
* query the latest SRPM for info * build SRPM and RPMs from SCM
Latest SRPM may be very different from SCM, for example when a major update is being prepared and built in a side tag. Incorrect BRs or arches could lead to incorrect test results.
I'm wondering why koji needs BRs/arches before building the SRPM, in other words: Why can't koji build the srpm (it rebuilds it now anyways) and then query it for arches and BRs? At worst, there is the following option:
Koji doesn't need BRs and arches, Koschei does.
Yes, typo, sorry. Everything here is about koschei. The question is still "why?"
* query the latest SRPM for info * build SRPM and RPMs from SCMLatest SRPM may be very different from SCM, for example when a major update is being prepared and built in a side tag. Incorrect BRs or arches could lead to incorrect test results.
... which is why I said "warn". Currently, it's a big red warning for failed builds even if it's fixed in scm (or by a repaired chroot). If koschei builds against SCM it would not fail in these cases. If SCM-built SRPM != last built SRPM koschei could warn (less red and more orange or yellow) about this.
Are you sure koschei even looks at side-tags? If it does it's clear the builds are against a different target, all the info is there.
I'd really like to see this implemented so I don't have to bump a release and do a build every time I fix FTBFS just to make that change propagate to Koschei. Is there anything I can help with?
