make install is not incremental for non-in-place builds

[riastradh]

What version of Racket are you using? 8.10

What program did you run?

./configure ...
gmake
gmake install
gmake install

What should have happened? Since I didn't rerun configure or change the content of any of the file inputs, gmake install should complete almost instantly without doing anything extra. (I tweaked some things outside Racket, which is what caused an automated process to rerun gmake install, but that's all.)

Instead, it seems to spend a long time recompiling all the Scheme code that was installed, even though it had already been compiled and I see no reason why it should have changed.

As a consequence, this makes incremental work on fixing build issues very slow and tedious.

Please include any other relevant details NetBSD 9

[samth]

Calling raco setup (which is the primary expensive thing make install does) is in fact incremental; it detects already compiled files and doesn't recompile them, and similar for documentation. That incrementality is not as fast as one might like, which is why it still takes a while, but it is very much there.

[riastradh]

Here's an excerpt from the log which makes me think it is recompiling:

Copying collects -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects
Copying share/pkgs -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/pkgs
Copying share -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket
Copying doc -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/doc/racket
  missing source path "doc", skipping...
Copying etc -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket
Rewriting configuration file at: /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket/config.rktd...
/home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket -X /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects -G /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket -N raco -l- setup --no-user
raco setup: ignoring compiled files, rebuilding from source...
 triggered by use of non-".zo" file
  path: /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects/compiler/private/cm-minimal.rkt

Am I reading this wrong? I'm particularly noticing this line: raco setup: ignoring compiled files, rebuilding from source...

(I can record the whole thing in a typescript if you like, just need to start it over again.)

[samth]

That does seem like it's recompiling but it shouldn't happen if you already re-ran raco setup. Could something be off about the timestamps of content of that file and it's corresponding compiled version?

[riastradh]

My guess -- without looking at anything more than the messages I quoted -- is that copying the files again is what changes the timestamp that raco setup is looking at. Does that sound plausible?

Here's more of the transcript, still had it in scrollback:

bin/zuo . install DESTDIR="/home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir" PLT_SETUP_OPTIONS=""
cp cs/c/racketcs /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket
strip /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket
cp start/starter-sh /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/starter-sh
cp cs/c/starter /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/starter
strip /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/starter
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d start/collects-path.rkt . /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/starter /home/riastradh/pkgsrc/current/pkg/share/racket/collects /home/riastradh/pkgsrc/current/pkg/etc/racket
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d cs/c/gen-system.rkt /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/system.rktd ta6nb ta6nb machine cs/c/ ""
cp cs/c/api.h /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/include/racket/racketcs.h
cp cs/c/boot.h /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/include/racket/racketcsboot.h
cp cs/c/ChezScheme/ta6nb/boot/ta6nb/scheme.h /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/include/racket/chezscheme.h
cd cs/c/repack && ar x ../ChezScheme/ta6nb/lz4/lib/liblz4.a
cd cs/c/repack && ar x ../rktio/librktio.a
cd cs/c/repack && ar x ../ChezScheme/ta6nb/boot/ta6nb/libkernel.a
ar rc cs/c/libracketcs.a cs/c/repack/rktio_error.o cs/c/repack/rktio_hash.o cs/c/repack/gc-011.o cs/c/repack/rktio_sha2.o cs/c/repack/rktio_syslog.o cs/c/repack/rktio_envvars.o cs/c/repack/rktio_wide.o cs/c/repack/random.o cs/c/repack/symbol.o cs/c/repack/lz4frame.o cs/c/repack/compress-io.o cs/c/repack/rktio_shellex.o cs/c/repack/thread.o cs/c/repack/prim5.o cs/c/repack/prim.o cs/c/repack/foreign.o cs/c/repack/rktio_network.o cs/c/repack/lz4.o cs/c/repack/rktio_convert.o cs/c/repack/expeditor.o cs/c/repack/gc-ocd.o cs/c/repack/fasl.o cs/c/repack/print.o cs/c/repack/rktio_sha1.o cs/c/repack/statics.o cs/c/repack/rktio_main.o cs/c/repack/scheme.o cs/c/repack/number.o cs/c/repack/gc-oce.o cs/c/repack/rktio_time.o cs/c/repack/i3le.o cs/c/repack/segment.o cs/c/repack/ffi.o cs/c/repack/lz4hc.o cs/c/repack/rktio_poll_set.o cs/c/repack/alloc.o cs/c/repack/rktio_fs_change.o cs/c/repack/flushcache.o cs/c/repack/rktio_flock.o cs/c/repack/stats.o cs/c/repack/schsig.o cs/c/repack/rktio_cpu.o cs/c/repack/schlib.o cs/c/repack/rktio_pipe.o cs/c/repack/rktio_fd.o cs/c/repack/gc-par.o cs/c/repack/rktio_ltps.o cs/c/repack/rktio_console.o cs/c/repack/rktio_fs.o cs/c/repack/io.o cs/c/repack/xxhash.o cs/c/repack/intern.o cs/c/repack/rktio_process.o cs/c/repack/vfasl.o cs/c/repack/rktio_dll.o cs/c/repack/rktio_signal.o cs/c/repack/new-io.o cs/c/repack/rktio_sleep.o cs/c/repack/gcwrapper.o cs/c/repack/rktio_file.o cs/c/boot.o
cp cs/c/libracketcs.a /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/libracketcs.a
strip -S /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/libracketcs.a
cp cs/c/gracketcs /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/gracket
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d start/collects-path.rkt . /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d start/collects-path.rkt . /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/gracket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d cs/c/add-terminator.rkt cs/c/petite-v.boot /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/petite.boot
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d cs/c/add-terminator.rkt cs/c/scheme-v.boot /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/scheme.boot
cs/c/raw_racketcs -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d cs/c/add-terminator.rkt cs/c/racket-v.boot /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket/racket.boot
/home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket -X ../collects -G ../etc -O info'@'compiler/cm -l- setup --boot setup-go.rkt cs/c/compiled ignored cs/c/ignored.d ../collects/setup/unixstyle-install.rkt make-install-copytree .. /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/pkgs /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/doc/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/include/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/lib/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/applications /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/man no
Copying collects -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects
Copying share/pkgs -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/pkgs
Copying share -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket
Copying doc -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/doc/racket
  missing source path "doc", skipping...
Copying etc -> /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket
Rewriting configuration file at: /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket/config.rktd...
/home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/bin/racket -X /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects -G /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/etc/racket -N raco -l- setup --no-user
raco setup: ignoring compiled files, rebuilding from source...
 triggered by use of non-".zo" file
  path: /home/riastradh/pkgsrc/current/work/lang/racket/work/.destdir/home/riastradh/pkgsrc/current/pkg/share/racket/collects/compiler/private/cm-minimal.rkt

[mflatt]

While raco setup is incremental, make install for a non-in-place install isn't incremental. That is, the title of the issue is correct.

I forget the details, but I think the "copying" step is implemented by removing things in the target directory first, instead of detecting when files it wants to copy are already in place and/or when ".zo" files in the destination should remain (as opposed to being removed because the corresponding source is no longer there, say).

[riastradh]

Maybe the files could be compiled in one step during make, and then just copied into the destdir in make install, rather than doing anything computationally-intensive in make install at all? That's how most software divides the configure/make/make install stages.

[juhp]

(Just adding that incremental building during install also wouldn't help (debugging) rpm builds, since rpmbuild removes all previously staged files in its %install phase anyway, so yeah (I expect it is a non-trivial raco change but) I would indeed also like to see what @riastradh suggests)

[mflatt]

There's a long history behind the current make versus make install split, but this is worth revisiting. Last year's overhaul of the build system (shifting to Zuo) should make a change much easier than previous times I considered making a change.