nx-libs icon indicating copy to clipboard operation
nx-libs copied to clipboard

Published 20 hours ago • verified publisher icon ArcticaProject

Random parallel build failures

Open apteryks opened this issue 1 year ago • 10 comments

Hi,

The build of nx-libs appears to suffer from non-deterministic build failures when parallelized (make -jN). Here's an example:

making all in programs/Xserver...
make[6]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
make[6]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/include...
making all in programs/Xserver/dix...
making all in programs/Xserver/os...
making all in programs/Xserver/mi...
making all in programs/Xserver/Xext...
making all in programs/Xserver/Xi...
making all in programs/Xserver/xkb...
making all in programs/Xserver/GL...
making all in programs/Xserver/dbe...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/record...
make[7]: Nothing to be done for 'all'.
make[7]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver/include
'
making all in programs/Xserver/randr...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/render...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/xfixes...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/damageext...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: *** No rule to make target 'all'.  Stop.
make[7]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver/os'
making all in programs/Xserver/miext/cw...
make[6]: *** [Makefile:682: os] Error 2
make[6]: *** Waiting for unfinished jobs....

It'd be nice if it worked normally. I also not that parallel build is disabled in the Debian rules file (--no-parallel).

apteryks avatar Mar 07 '24 20:03 apteryks

You ouput is unclear to me. What subdir exactly fails?

Is it damageext?

uli42 avatar Apr 08 '24 12:04 uli42

It's not clear to me either, but that's what I see! It does look like the error is triggered from the damageext directory.

apteryks avatar Apr 10 '24 18:04 apteryks

Just stumbled over this in nx-X11/programs/Xserver/Imakefile:

#if HasParallelMake
MakeMutex($(NXAGENTDIRS) $(NXCOMPEXT) $(NXAGENTOBJS) $(NXAGENTLIBS) $(NXCOMPEXTSYSLIBS) $(NXAGENTSYSLIBS))
#endif

So you probably have to run the topdir makefile with IMAKE_DEFINES=-DHasParallelMake=1

uli42 avatar May 09 '24 23:05 uli42

Hm, I tried that, but it didn't seem to make any difference:

make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include/GL'
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include'
including in ./config...
make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config'
including in config/cf...
make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf'
make[5]: Nothing to be done for 'includes'.
make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf'
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config'
including in ./programs...
make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
including in programs/Xserver...
make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
Makefile:539: *** missing separator.  Stop.
make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[4]: *** [Makefile:615: includes] Error 2
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
make[3]: *** [xmakefile:674: includes] Error 2
make[3]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11'
make[2]: *** [Makefile:102: includes] Error 2
make[2]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11'
make[1]: *** [Makefile:148: build-env] Error 2
make[1]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source'
make: *** [Makefile:220: build] Error 2
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1") exit-status: 2 term-signal: #f stop-signal: #f> 
phase `build' failed after 5.9 seconds
command "make" "-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1" failed with status 2
builder for `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed with exit code 1
build of /gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv failed
View build log at '/var/log/guix/drvs/zi/k74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv.gz'.
guix build: error: build of `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed

real    0m42.514s
user    0m6.178s
sys     0m0.594s

apteryks avatar May 19 '24 01:05 apteryks

I see "./configure" in the output. That is used for some components. Of course that is not synced with the imake build. So I think you have to modify the main makefile to introduce some "barriers" First compile the automake parts and then the imake parts (xserver). Be aware that nxcomp and nxcompshad are required by the other parts. And the server depends on both and libNX_x11.

If you are wiling to help there's an automake branch that needs some polishing. It should be findable in the pull-requests. Let me know if you don't find it.

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

Uli

Apteryks @.***> schrieb am So., 19. Mai 2024, 03:11:

Hm, I tried that, but it didn't seem to make any difference:

make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include/GL' make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include' including in ./config... make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config' including in config/cf... make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf' make[5]: Nothing to be done for 'includes'. make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf' make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config' including in ./programs... make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs' including in programs/Xserver... make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver' Makefile:539: *** missing separator. Stop. make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver' make[4]: *** [Makefile:615: includes] Error 2 make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs' make[3]: *** [xmakefile:674: includes] Error 2 make[3]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11' make[2]: *** [Makefile:102: includes] Error 2 make[2]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11' make[1]: *** [Makefile:148: build-env] Error 2 make[1]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source' make: *** [Makefile:220: build] Error 2 error: in phase 'build': uncaught exception: %exception #<&invoke-error program: "make" arguments: ("-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1") exit-status: 2 term-signal: #f stop-signal: #f> phase build' failed after 5.9 seconds command "make" "-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1" failed with status 2 builder for /gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed with exit code 1 build of /gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv failed View build log at '/var/log/guix/drvs/zi/k74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv.gz'. guix build: error: build of `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed

real 0m42.514s user 0m6.178s sys 0m0.594s

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2119049992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZBVX4SJEFQJGEEHA4DZC734PAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGA2DSOJZGI . You are receiving this because you commented.Message ID: @.***>

uli42 avatar May 19 '24 10:05 uli42

If you are wiling to help there's an automake branch that needs some polishing. It should be findable in the pull-requests. Let me know if you don't find it.

That's tempting but am I bit swamped with things to do at the moment. I'll try to remember revisiting this in the future!

apteryks avatar May 21 '24 23:05 apteryks

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

"Was" makes it sound like the migration of the build system to Meson already happened? Is that so?

apteryks avatar May 21 '24 23:05 apteryks

No, "was" is subjunctive, like "If I were rich" ;-)

Uli

On Wed, May 22, 2024 at 1:58 AM Apteryks @.***> wrote:

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

"Was" makes it sound like the migration of the build system to Meson already happened? Is that so?

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2123611828, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZCFZI4T3U4SOUZODDTZDPNTTAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGYYTCOBSHA . You are receiving this because you commented.Message ID: @.***>

uli42 avatar May 22 '24 10:05 uli42

Ah, I see :-) I guess I'll look at that Autotools branch at some point; it'd be a nice improvement over a build system whose default assumption is that make doesn't support parallel builds ^^'.

apteryks avatar May 22 '24 12:05 apteryks

;-)

Here are the respective PRs: https://github.com/ArcticaProject/nx-libs/pull/947 https://github.com/ArcticaProject/nx-libs/pull/946

On Wed, May 22, 2024 at 2:33 PM Apteryks @.***> wrote:

Ah, I see :-) I guess I'll look at that Autotools branch at some point; it'd be a nice improvement over a build system whose default assumption is that make doesn't support parallel builds ^^'.

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2124679613, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZA65DCW5IQUFD7P6QLZDSGAJAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRUGY3TSNRRGM . You are receiving this because you commented.Message ID: @.***>

uli42 avatar May 22 '24 18:05 uli42